[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:45:14 dist-test-slave-gmjp krb5kdc[1768](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:45:14 dist-test-slave-gmjp krb5kdc[1768](info): set up 2 sockets
Aug 06 02:45:14 dist-test-slave-gmjp krb5kdc[1768](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260806 02:45:15.215739 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.290s user 0.007s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:45:15 dist-test-slave-gmjp krb5kdc[1768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:45:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
I20260806 02:45:15.357383 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:42365
--webserver_interface=127.1.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:43177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:42365
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:15.475611 1785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:15.475955 1785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:15.476039 1785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:15.480139 1785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:15.480259 1785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:15.480295 1785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:15.480333 1785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:15.480365 1785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:15.485854 1785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:43177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:42365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:42365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:15.487054 1785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:15.488195 1785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:45:15.497619 1785 server_base.cc:1034] running on GCE node
W20260806 02:45:15.497556 1792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:15.497558 1796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:15.497556 1794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:15.498540 1785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:15.499748 1785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:15.500931 1785 hybrid_clock.cc:648] HybridClock initialized: now 1785984315500857 us; error 94 us; skew 500 ppm
Aug 06 02:45:15 dist-test-slave-gmjp krb5kdc[1768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984315, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:15.504276 1785 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:45:15.505394 1785 webserver.cc:460] Webserver started at http://127.1.183.190:38469/ using document root <none> and password file <none>
I20260806 02:45:15.506059 1785 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:15.506131 1785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:15.506407 1785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:15.508934 1785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "8556538f5ee94a73b4aed1fb148efd7c"
format_stamp: "Formatted at 2026-08-06 02:45:15 on dist-test-slave-gmjp"
server_key: "b470270c1aba56f859b8d0d542d8c888"
server_key_iv: "ed12ce225cf20f5ffa614f7766c34d43"
server_key_version: "encryptionkey@0"
I20260806 02:45:15.509591 1785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "8556538f5ee94a73b4aed1fb148efd7c"
format_stamp: "Formatted at 2026-08-06 02:45:15 on dist-test-slave-gmjp"
server_key: "b470270c1aba56f859b8d0d542d8c888"
server_key_iv: "ed12ce225cf20f5ffa614f7766c34d43"
server_key_version: "encryptionkey@0"
I20260806 02:45:15.514107 1785 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260806 02:45:15.517127 1805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:15.518550 1785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:45:15.518721 1785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "8556538f5ee94a73b4aed1fb148efd7c"
format_stamp: "Formatted at 2026-08-06 02:45:15 on dist-test-slave-gmjp"
server_key: "b470270c1aba56f859b8d0d542d8c888"
server_key_iv: "ed12ce225cf20f5ffa614f7766c34d43"
server_key_version: "encryptionkey@0"
I20260806 02:45:15.518863 1785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:15.536536 1785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:15.539871 1785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:15.540107 1785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:15.547731 1785 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:42365
I20260806 02:45:15.547786 1882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:42365 every 8 connection(s)
I20260806 02:45:15.548808 1785 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:15.551843 1883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:15.555945 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 1785
I20260806 02:45:15.556082 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:45:15.557416 1758 external_mini_cluster.cc:1428] Setting key 9e5a0d2630907cd27392faff68f2e2a2
I20260806 02:45:15.558133 1883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8556538f5ee94a73b4aed1fb148efd7c: Bootstrap starting.
I20260806 02:45:15.561082 1883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8556538f5ee94a73b4aed1fb148efd7c: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:15.562039 1883 log.cc:824] T 00000000000000000000000000000000 P 8556538f5ee94a73b4aed1fb148efd7c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:15.564517 1883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8556538f5ee94a73b4aed1fb148efd7c: No bootstrap required, opened a new log
I20260806 02:45:15.569258 1883 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8556538f5ee94a73b4aed1fb148efd7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8556538f5ee94a73b4aed1fb148efd7c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 42365 } }
I20260806 02:45:15.569511 1883 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8556538f5ee94a73b4aed1fb148efd7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:15.569619 1883 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8556538f5ee94a73b4aed1fb148efd7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8556538f5ee94a73b4aed1fb148efd7c, State: Initialized, Role: FOLLOWER
I20260806 02:45:15.570169 1883 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8556538f5ee94a73b4aed1fb148efd7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8556538f5ee94a73b4aed1fb148efd7c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 42365 } }
I20260806 02:45:15.570356 1883 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8556538f5ee94a73b4aed1fb148efd7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:15.570434 1883 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8556538f5ee94a73b4aed1fb148efd7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:15.570540 1883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8556538f5ee94a73b4aed1fb148efd7c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:15.571919 1883 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8556538f5ee94a73b4aed1fb148efd7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8556538f5ee94a73b4aed1fb148efd7c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 42365 } }
I20260806 02:45:15.572382 1883 leader_election.cc:302] T 00000000000000000000000000000000 P 8556538f5ee94a73b4aed1fb148efd7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8556538f5ee94a73b4aed1fb148efd7c; no voters:
I20260806 02:45:15.572784 1883 leader_election.cc:288] T 00000000000000000000000000000000 P 8556538f5ee94a73b4aed1fb148efd7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:15.572880 1894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8556538f5ee94a73b4aed1fb148efd7c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:15.573163 1894 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8556538f5ee94a73b4aed1fb148efd7c [term 1 LEADER]: Becoming Leader. State: Replica: 8556538f5ee94a73b4aed1fb148efd7c, State: Running, Role: LEADER
I20260806 02:45:15.573498 1894 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8556538f5ee94a73b4aed1fb148efd7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8556538f5ee94a73b4aed1fb148efd7c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 42365 } }
I20260806 02:45:15.574275 1883 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8556538f5ee94a73b4aed1fb148efd7c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:15.575260 1900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8556538f5ee94a73b4aed1fb148efd7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8556538f5ee94a73b4aed1fb148efd7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8556538f5ee94a73b4aed1fb148efd7c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 42365 } } }
I20260806 02:45:15.575392 1900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8556538f5ee94a73b4aed1fb148efd7c [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:15.575766 1901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8556538f5ee94a73b4aed1fb148efd7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8556538f5ee94a73b4aed1fb148efd7c. Latest consensus state: current_term: 1 leader_uuid: "8556538f5ee94a73b4aed1fb148efd7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8556538f5ee94a73b4aed1fb148efd7c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 42365 } } }
I20260806 02:45:15.575886 1901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8556538f5ee94a73b4aed1fb148efd7c [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:15.576259 1908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:15.580226 1908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 06 02:45:15 dist-test-slave-gmjp krb5kdc[1768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:15.587159 1908 catalog_manager.cc:1347] Generated new cluster ID: daf67adfa054452a8140b576204a5a2b
I20260806 02:45:15.587234 1908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:15.590476 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:15.560470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46998 (local address 127.1.183.190:42365)
0806 02:45:15.561005 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:15.561027 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:15.562245 (+ 1218us) server_negotiation.cc:407] Connection header received
0806 02:45:15.568835 (+ 6590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:15.568860 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:15.569268 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:15.569685 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:15.573227 (+ 3542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:15.574847 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:15.576741 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:15.577468 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:15.577639 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:15.582969 (+ 5330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:15.582977 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:15.582995 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:15.583034 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:15.585462 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:15.586262 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:15.586272 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:15.586280 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:15.586492 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:15.586957 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:15.586961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:15.586963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:15.587546 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:15.587812 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:15.589449 (+ 1637us) server_negotiation.cc:299] Negotiation successful
0806 02:45:15.589700 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":156,"threads_started":1}
I20260806 02:45:15.620069 1908 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:15.621127 1908 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:15.632073 1908 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8556538f5ee94a73b4aed1fb148efd7c: Generated new TSK 0
I20260806 02:45:15.633095 1908 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
I20260806 02:45:15.671662 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.129:0
--local_ip_for_outbound_sockets=127.1.183.129
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:42365
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:43177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:15.796602 1925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:15.796936 1925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:15.796985 1925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:15.801376 1925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:15.801455 1925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:15.801548 1925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.129
I20260806 02:45:15.807531 1925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:43177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:42365
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.129
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:15.808853 1925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:15.809973 1925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:15.820117 1931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:15.820240 1930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:15.820365 1925 server_base.cc:1034] running on GCE node
W20260806 02:45:15.820114 1933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:15.820973 1925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:15.821691 1925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:15.822878 1925 hybrid_clock.cc:648] HybridClock initialized: now 1785984315822839 us; error 55 us; skew 500 ppm
Aug 06 02:45:15 dist-test-slave-gmjp krb5kdc[1768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984315, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:15.825919 1925 init.cc:377] Logged in from keytab as kudu/127.1.183.129@KRBTEST.COM (short username kudu)
I20260806 02:45:15.826850 1925 webserver.cc:460] Webserver started at http://127.1.183.129:45469/ using document root <none> and password file <none>
I20260806 02:45:15.827407 1925 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:15.827467 1925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:15.827638 1925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:15.829871 1925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-0/data/instance:
uuid: "8900a678e4c3414c87f91d0910c09646"
format_stamp: "Formatted at 2026-08-06 02:45:15 on dist-test-slave-gmjp"
server_key: "00b716ba33ca420084e4dadcabb31fd9"
server_key_iv: "93a3c2a20a6031ea55624425b4371b5e"
server_key_version: "encryptionkey@0"
I20260806 02:45:15.830457 1925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance:
uuid: "8900a678e4c3414c87f91d0910c09646"
format_stamp: "Formatted at 2026-08-06 02:45:15 on dist-test-slave-gmjp"
server_key: "00b716ba33ca420084e4dadcabb31fd9"
server_key_iv: "93a3c2a20a6031ea55624425b4371b5e"
server_key_version: "encryptionkey@0"
I20260806 02:45:15.834443 1925 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260806 02:45:15.837145 1943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:15.838469 1925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 02:45:15.838578 1925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-0/wal
uuid: "8900a678e4c3414c87f91d0910c09646"
format_stamp: "Formatted at 2026-08-06 02:45:15 on dist-test-slave-gmjp"
server_key: "00b716ba33ca420084e4dadcabb31fd9"
server_key_iv: "93a3c2a20a6031ea55624425b4371b5e"
server_key_version: "encryptionkey@0"
I20260806 02:45:15.838675 1925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:15.852501 1925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:15.855978 1925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:15.856200 1925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:15.856863 1925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:15.857733 1925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:15.857782 1925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:15.857824 1925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:15.857841 1925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:15.868287 1925 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.129:36523
I20260806 02:45:15.868304 2110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.129:36523 every 8 connection(s)
I20260806 02:45:15.869280 1925 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
I20260806 02:45:15.870301 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 1925
I20260806 02:45:15.870393 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance
I20260806 02:45:15.870662 1758 external_mini_cluster.cc:1428] Setting key 2a9d3c9019e0682aaecef0f6819935f3
Aug 06 02:45:15 dist-test-slave-gmjp krb5kdc[1768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984315, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.130@KRBTEST.COM" created.
I20260806 02:45:15.886615 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:15.871528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:45595 (local address 127.1.183.190:42365)
0806 02:45:15.871670 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:15.871675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:15.872723 (+ 1048us) server_negotiation.cc:407] Connection header received
0806 02:45:15.874292 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:15.874298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:15.874377 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:15.874505 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:15.876277 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:15.876871 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:15.878737 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:15.879195 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:15.879360 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:15.882207 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:15.882211 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:15.882214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:15.882246 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:15.883800 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:15.884521 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:15.884524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:15.884525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:15.884650 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:15.885115 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:15.885118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:15.885119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:15.885458 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:15.885581 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:15.886221 (+ 640us) server_negotiation.cc:299] Negotiation successful
0806 02:45:15.886404 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 02:45:15.887624 2116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:15.871978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:42365 (local address 127.1.183.129:45595)
0806 02:45:15.872575 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:15.872620 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:15.873985 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:15.874768 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:15.874778 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:15.875347 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:15.876103 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:15.876128 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:15.877014 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:15.877020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:15.878597 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:15.878608 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:15.879361 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:15.879364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:15.879509 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:15.880210 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:15.880230 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:15.882000 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:15.883934 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:15.883939 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:15.883954 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:15.884323 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:15.884774 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:15.884776 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:15.884778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:15.884945 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:15.885599 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:15.885607 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:15.885926 (+ 319us) client_negotiation.cc:769] Sending connection context
0806 02:45:15.886155 (+ 229us) client_negotiation.cc:240] Negotiation successful
0806 02:45:15.886557 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":198,"threads_started":1}
I20260806 02:45:15.889159 2113 heartbeater.cc:347] Connected to a master server at 127.1.183.190:42365
I20260806 02:45:15.889446 2113 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:15.890029 2113 heartbeater.cc:511] Master 127.1.183.190:42365 requested a full tablet report, sending...
I20260806 02:45:15.891742 1830 ts_manager.cc:194] Registered new tserver with Master: 8900a678e4c3414c87f91d0910c09646 (127.1.183.129:36523)
I20260806 02:45:15.892984 1830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.129@KRBTEST.COM'} at 127.1.183.129:45595
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
I20260806 02:45:15.903731 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.130:0
--local_ip_for_outbound_sockets=127.1.183.130
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:42365
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:43177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:16.021890 2119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:16.022174 2119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:16.022260 2119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:16.026410 2119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:16.026489 2119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:16.026557 2119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.130
I20260806 02:45:16.032241 2119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:43177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:42365
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.130
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:16.033452 2119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:16.034543 2119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:16.044708 2127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:16.044714 2130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:16.044714 2128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:16.045157 2119 server_base.cc:1034] running on GCE node
I20260806 02:45:16.045578 2119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:16.046265 2119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:16.047490 2119 hybrid_clock.cc:648] HybridClock initialized: now 1785984316047473 us; error 31 us; skew 500 ppm
Aug 06 02:45:16 dist-test-slave-gmjp krb5kdc[1768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984316, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:16.050551 2119 init.cc:377] Logged in from keytab as kudu/127.1.183.130@KRBTEST.COM (short username kudu)
I20260806 02:45:16.051541 2119 webserver.cc:460] Webserver started at http://127.1.183.130:41641/ using document root <none> and password file <none>
I20260806 02:45:16.052187 2119 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:16.052289 2119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:16.052531 2119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:16.054723 2119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-1/data/instance:
uuid: "058b1b10fc28460597a847cedf3a91a5"
format_stamp: "Formatted at 2026-08-06 02:45:16 on dist-test-slave-gmjp"
server_key: "a2044127aed85bd3b94cb780447ac97e"
server_key_iv: "7404511dfe28c517625c05eba47bdd95"
server_key_version: "encryptionkey@0"
I20260806 02:45:16.055375 2119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance:
uuid: "058b1b10fc28460597a847cedf3a91a5"
format_stamp: "Formatted at 2026-08-06 02:45:16 on dist-test-slave-gmjp"
server_key: "a2044127aed85bd3b94cb780447ac97e"
server_key_iv: "7404511dfe28c517625c05eba47bdd95"
server_key_version: "encryptionkey@0"
I20260806 02:45:16.060024 2119 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 02:45:16.062829 2141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:16.064299 2119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:16.064450 2119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-1/wal
uuid: "058b1b10fc28460597a847cedf3a91a5"
format_stamp: "Formatted at 2026-08-06 02:45:16 on dist-test-slave-gmjp"
server_key: "a2044127aed85bd3b94cb780447ac97e"
server_key_iv: "7404511dfe28c517625c05eba47bdd95"
server_key_version: "encryptionkey@0"
I20260806 02:45:16.064585 2119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:16.077050 2119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:16.080067 2119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:16.080304 2119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:16.080930 2119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:16.081796 2119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:16.081883 2119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:16.081970 2119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:16.082016 2119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:16.091398 2119 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.130:40173
I20260806 02:45:16.091416 2305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.130:40173 every 8 connection(s)
I20260806 02:45:16.092433 2119 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
I20260806 02:45:16.101729 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 2119
I20260806 02:45:16.101831 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance
I20260806 02:45:16.102213 1758 external_mini_cluster.cc:1428] Setting key 882e6b0d84f271f993669daa6e50e354
Aug 06 02:45:16 dist-test-slave-gmjp krb5kdc[1768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984316, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:16.107563 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:16.094420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:56969 (local address 127.1.183.190:42365)
0806 02:45:16.094557 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:16.094561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:16.095398 (+ 837us) server_negotiation.cc:407] Connection header received
0806 02:45:16.096918 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:16.096921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:16.096992 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:16.097072 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:16.098324 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:16.098876 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:16.100075 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:16.100389 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:16.100501 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:16.103268 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:16.103271 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:16.103274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:16.103300 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:16.105159 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:16.105779 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:16.105786 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:16.105790 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:16.105897 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:16.106358 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:16.106362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:16.106364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:16.106569 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:16.106683 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:16.107251 (+ 568us) server_negotiation.cc:299] Negotiation successful
0806 02:45:16.107392 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 02:45:16.108817 2310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:16.094773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:42365 (local address 127.1.183.130:56969)
0806 02:45:16.095240 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:16.095286 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:16.096675 (+ 1389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:16.097250 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:16.097258 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:16.097639 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:16.098155 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:16.098170 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:16.099023 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:16.099027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:16.099935 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:16.099945 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:16.100566 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:16.100571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:16.100746 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:16.101328 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:16.101346 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:16.103105 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:16.105321 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:16.105327 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:16.105344 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:16.105669 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:16.106000 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:16.106002 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:16.106004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:16.106225 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:16.106683 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:16.106692 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:16.107001 (+ 309us) client_negotiation.cc:769] Sending connection context
0806 02:45:16.107210 (+ 209us) client_negotiation.cc:240] Negotiation successful
0806 02:45:16.107597 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":112,"threads_started":1}
I20260806 02:45:16.110215 2306 heartbeater.cc:347] Connected to a master server at 127.1.183.190:42365
I20260806 02:45:16.110502 2306 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:16.111058 2306 heartbeater.cc:511] Master 127.1.183.190:42365 requested a full tablet report, sending...
I20260806 02:45:16.112432 1830 ts_manager.cc:194] Registered new tserver with Master: 058b1b10fc28460597a847cedf3a91a5 (127.1.183.130:40173)
I20260806 02:45:16.113076 1830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.130@KRBTEST.COM'} at 127.1.183.130:56969
WARNING: no policy specified for kudu/127.1.183.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
I20260806 02:45:16.134749 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.131:0
--local_ip_for_outbound_sockets=127.1.183.131
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:42365
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:43177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:16.259266 2314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:16.259651 2314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:16.259737 2314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:16.264994 2314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:16.265084 2314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:16.265185 2314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.131
I20260806 02:45:16.271097 2314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:43177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:42365
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.131
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:16.272447 2314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:16.273551 2314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:16.283916 2320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:16.283937 2326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:16.283934 2321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:16.284281 2314 server_base.cc:1034] running on GCE node
I20260806 02:45:16.284734 2314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:16.285463 2314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:16.286640 2314 hybrid_clock.cc:648] HybridClock initialized: now 1785984316286611 us; error 41 us; skew 500 ppm
Aug 06 02:45:16 dist-test-slave-gmjp krb5kdc[1768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984316, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:16.289669 2314 init.cc:377] Logged in from keytab as kudu/127.1.183.131@KRBTEST.COM (short username kudu)
I20260806 02:45:16.290571 2314 webserver.cc:460] Webserver started at http://127.1.183.131:37543/ using document root <none> and password file <none>
I20260806 02:45:16.291128 2314 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:16.291188 2314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:16.291365 2314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:16.293563 2314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-2/data/instance:
uuid: "0df894f8a31f4db0927f097b1ec1468d"
format_stamp: "Formatted at 2026-08-06 02:45:16 on dist-test-slave-gmjp"
server_key: "b4e2d8c0fb9224aa0fe3b8c81f31f93c"
server_key_iv: "66ebed4ae8b8aa44a1151c0355a84020"
server_key_version: "encryptionkey@0"
I20260806 02:45:16.294138 2314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance:
uuid: "0df894f8a31f4db0927f097b1ec1468d"
format_stamp: "Formatted at 2026-08-06 02:45:16 on dist-test-slave-gmjp"
server_key: "b4e2d8c0fb9224aa0fe3b8c81f31f93c"
server_key_iv: "66ebed4ae8b8aa44a1151c0355a84020"
server_key_version: "encryptionkey@0"
I20260806 02:45:16.298243 2314 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:45:16.300837 2338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:16.302245 2314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:16.302408 2314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-2/wal
uuid: "0df894f8a31f4db0927f097b1ec1468d"
format_stamp: "Formatted at 2026-08-06 02:45:16 on dist-test-slave-gmjp"
server_key: "b4e2d8c0fb9224aa0fe3b8c81f31f93c"
server_key_iv: "66ebed4ae8b8aa44a1151c0355a84020"
server_key_version: "encryptionkey@0"
I20260806 02:45:16.302542 2314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:16.323958 2314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:16.327600 2314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:16.327859 2314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:16.328590 2314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:16.329608 2314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:16.329715 2314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:16.329811 2314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:16.329860 2314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:16.341535 2314 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.131:42729
I20260806 02:45:16.341558 2499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.131:42729 every 8 connection(s)
I20260806 02:45:16.342607 2314 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
I20260806 02:45:16.342811 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 2314
I20260806 02:45:16.342949 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance
I20260806 02:45:16.343333 1758 external_mini_cluster.cc:1428] Setting key 9ec8f2ead1b80e8025c992e2351bd316
Aug 06 02:45:16 dist-test-slave-gmjp krb5kdc[1768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984316, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:16.357316 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:16.345050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:41555 (local address 127.1.183.190:42365)
0806 02:45:16.345307 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:16.345311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:16.346274 (+ 963us) server_negotiation.cc:407] Connection header received
0806 02:45:16.347680 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:16.347683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:16.347741 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:16.347823 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:16.349167 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:16.349694 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:16.350906 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:16.351121 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:16.351207 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:16.353869 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:16.353872 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:16.353875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:16.353925 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:16.355243 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:16.355729 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:16.355731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:16.355733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:16.355827 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:16.356142 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:16.356144 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:16.356145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:16.356377 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:16.356457 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:16.357026 (+ 569us) server_negotiation.cc:299] Negotiation successful
0806 02:45:16.357136 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260806 02:45:16.358278 2502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:16.345554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:42365 (local address 127.1.183.131:41555)
0806 02:45:16.346113 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:16.346167 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:16.347468 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:16.347960 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:16.347966 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:16.348410 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:16.348995 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:16.349016 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:16.349856 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:16.349859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:16.350771 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:16.350781 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:16.351286 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:16.351289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:16.351448 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:16.352114 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:16.352132 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:16.353696 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:16.355353 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:16.355359 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:16.355376 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:16.355643 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:16.355923 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:16.355925 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:16.355926 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:16.356062 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:16.356470 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:16.356475 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:16.356730 (+ 255us) client_negotiation.cc:769] Sending connection context
0806 02:45:16.356912 (+ 182us) client_negotiation.cc:240] Negotiation successful
0806 02:45:16.357288 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":145,"threads_started":1}
I20260806 02:45:16.359421 2500 heartbeater.cc:347] Connected to a master server at 127.1.183.190:42365
I20260806 02:45:16.359661 2500 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:16.360224 2500 heartbeater.cc:511] Master 127.1.183.190:42365 requested a full tablet report, sending...
I20260806 02:45:16.361261 1830 ts_manager.cc:194] Registered new tserver with Master: 0df894f8a31f4db0927f097b1ec1468d (127.1.183.131:42729)
I20260806 02:45:16.361840 1830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.131@KRBTEST.COM'} at 127.1.183.131:41555
I20260806 02:45:16.370473 1758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:45:16 dist-test-slave-gmjp krb5kdc[1768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984316, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:45:16 dist-test-slave-gmjp krb5kdc[1768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984316, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.183.129@KRBTEST.COM
I20260806 02:45:16.399290 2508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:16.389456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56446 (local address 127.1.183.129:36523)
0806 02:45:16.389755 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:16.389765 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:16.389871 (+ 106us) server_negotiation.cc:407] Connection header received
0806 02:45:16.390087 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:16.390094 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:16.390299 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:16.390468 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:16.391172 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:16.391944 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:16.392864 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:16.393261 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:16.393379 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:16.395890 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:16.395904 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:16.395922 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:16.395948 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:16.397541 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:16.397966 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:16.397971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:16.397973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:16.398059 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:16.398282 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:16.398284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:16.398285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:16.398563 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:16.398734 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:16.398894 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 02:45:16.399066 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":72,"threads_started":1}
W20260806 02:45:16.400501 1981 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56446
Aug 06 02:45:16 dist-test-slave-gmjp krb5kdc[1768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:45:16 dist-test-slave-gmjp krb5kdc[1768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.129@KRBTEST.COM
I20260806 02:45:16.429495 2508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:16.420482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56460 (local address 127.1.183.129:36523)
0806 02:45:16.420673 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:16.420677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:16.420768 (+ 91us) server_negotiation.cc:407] Connection header received
0806 02:45:16.420885 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:16.420888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:16.420937 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:16.421017 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:16.421760 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:16.422373 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:16.423234 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:16.423479 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:16.423603 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:16.426309 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:16.426313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:16.426315 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:16.426340 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:16.427731 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:16.428216 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:16.428220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:16.428221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:16.428297 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:16.428576 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:16.428579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:16.428581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:16.428770 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:16.428940 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:16.429178 (+ 238us) server_negotiation.cc:299] Negotiation successful
0806 02:45:16.429313 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 02:45:16.430639 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 1925
I20260806 02:45:16.444223 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 2119
I20260806 02:45:16.455520 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 2314
I20260806 02:45:16.465106 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 1785
2026-08-06T02:45:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2580 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:45:16 dist-test-slave-gmjp krb5kdc[2518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:45:16 dist-test-slave-gmjp krb5kdc[2518](info): set up 2 sockets
Aug 06 02:45:16 dist-test-slave-gmjp krb5kdc[2518](info): commencing operation
krb5kdc: starting...
W20260806 02:45:17.704875 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.208s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:45:17 dist-test-slave-gmjp krb5kdc[2518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:45:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
I20260806 02:45:17.843546 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:44933
--webserver_interface=127.1.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:44785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:44933
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:17.961266 2537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:17.961612 2537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:17.961687 2537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:17.965924 2537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:17.966030 2537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:17.966068 2537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:17.966104 2537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:17.966145 2537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:17.971659 2537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:44785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:44933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:44933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:17.972941 2537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:17.974045 2537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:17.983003 2542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:17.983062 2543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:17.982997 2549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:17.983428 2537 server_base.cc:1034] running on GCE node
I20260806 02:45:17.984019 2537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:17.985291 2537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:17.986476 2537 hybrid_clock.cc:648] HybridClock initialized: now 1785984317986462 us; error 27 us; skew 500 ppm
Aug 06 02:45:17 dist-test-slave-gmjp krb5kdc[2518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984317, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:17.989387 2537 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:45:17.990361 2537 webserver.cc:460] Webserver started at http://127.1.183.190:32905/ using document root <none> and password file <none>
I20260806 02:45:17.990972 2537 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:17.991066 2537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:17.991305 2537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:17.993487 2537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "eacddc8753ad41808da2a57f2f5a986b"
format_stamp: "Formatted at 2026-08-06 02:45:17 on dist-test-slave-gmjp"
server_key: "e9fe43164624a3ddcf92cb2cd764d78a"
server_key_iv: "a99f21e77a898b9228bff9c29f02c778"
server_key_version: "encryptionkey@0"
I20260806 02:45:17.994073 2537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "eacddc8753ad41808da2a57f2f5a986b"
format_stamp: "Formatted at 2026-08-06 02:45:17 on dist-test-slave-gmjp"
server_key: "e9fe43164624a3ddcf92cb2cd764d78a"
server_key_iv: "a99f21e77a898b9228bff9c29f02c778"
server_key_version: "encryptionkey@0"
I20260806 02:45:17.998293 2537 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:45:18.001020 2559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:18.002477 2537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:18.002668 2537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "eacddc8753ad41808da2a57f2f5a986b"
format_stamp: "Formatted at 2026-08-06 02:45:17 on dist-test-slave-gmjp"
server_key: "e9fe43164624a3ddcf92cb2cd764d78a"
server_key_iv: "a99f21e77a898b9228bff9c29f02c778"
server_key_version: "encryptionkey@0"
I20260806 02:45:18.002811 2537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:18.019465 2537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:18.023213 2537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:18.023451 2537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:18.031478 2537 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:44933
I20260806 02:45:18.031507 2647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:44933 every 8 connection(s)
I20260806 02:45:18.032562 2537 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:18.035470 2648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:18.041424 2648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eacddc8753ad41808da2a57f2f5a986b: Bootstrap starting.
I20260806 02:45:18.041540 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 2537
I20260806 02:45:18.041638 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:45:18.041913 1758 external_mini_cluster.cc:1428] Setting key c3d4693c6c0e89f7e5b8e106fd4efda0
I20260806 02:45:18.044332 2648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eacddc8753ad41808da2a57f2f5a986b: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:18.045120 2648 log.cc:824] T 00000000000000000000000000000000 P eacddc8753ad41808da2a57f2f5a986b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:18.047058 2648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eacddc8753ad41808da2a57f2f5a986b: No bootstrap required, opened a new log
I20260806 02:45:18.050961 2648 raft_consensus.cc:348] T 00000000000000000000000000000000 P eacddc8753ad41808da2a57f2f5a986b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eacddc8753ad41808da2a57f2f5a986b" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 44933 } }
I20260806 02:45:18.051196 2648 raft_consensus.cc:374] T 00000000000000000000000000000000 P eacddc8753ad41808da2a57f2f5a986b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Aug 06 02:45:18 dist-test-slave-gmjp krb5kdc[2518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:18.051265 2648 raft_consensus.cc:729] T 00000000000000000000000000000000 P eacddc8753ad41808da2a57f2f5a986b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eacddc8753ad41808da2a57f2f5a986b, State: Initialized, Role: FOLLOWER
I20260806 02:45:18.051759 2648 consensus_queue.cc:260] T 00000000000000000000000000000000 P eacddc8753ad41808da2a57f2f5a986b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eacddc8753ad41808da2a57f2f5a986b" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 44933 } }
I20260806 02:45:18.051908 2648 raft_consensus.cc:388] T 00000000000000000000000000000000 P eacddc8753ad41808da2a57f2f5a986b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:18.051975 2648 raft_consensus.cc:482] T 00000000000000000000000000000000 P eacddc8753ad41808da2a57f2f5a986b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:18.052075 2648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eacddc8753ad41808da2a57f2f5a986b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:18.053265 2648 raft_consensus.cc:504] T 00000000000000000000000000000000 P eacddc8753ad41808da2a57f2f5a986b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eacddc8753ad41808da2a57f2f5a986b" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 44933 } }
I20260806 02:45:18.053617 2648 leader_election.cc:302] T 00000000000000000000000000000000 P eacddc8753ad41808da2a57f2f5a986b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eacddc8753ad41808da2a57f2f5a986b; no voters:
I20260806 02:45:18.053936 2648 leader_election.cc:288] T 00000000000000000000000000000000 P eacddc8753ad41808da2a57f2f5a986b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:18.054060 2654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eacddc8753ad41808da2a57f2f5a986b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:18.054291 2654 raft_consensus.cc:686] T 00000000000000000000000000000000 P eacddc8753ad41808da2a57f2f5a986b [term 1 LEADER]: Becoming Leader. State: Replica: eacddc8753ad41808da2a57f2f5a986b, State: Running, Role: LEADER
I20260806 02:45:18.054610 2654 consensus_queue.cc:237] T 00000000000000000000000000000000 P eacddc8753ad41808da2a57f2f5a986b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eacddc8753ad41808da2a57f2f5a986b" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 44933 } }
I20260806 02:45:18.055349 2648 sys_catalog.cc:564] T 00000000000000000000000000000000 P eacddc8753ad41808da2a57f2f5a986b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:18.056631 2655 sys_catalog.cc:455] T 00000000000000000000000000000000 P eacddc8753ad41808da2a57f2f5a986b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eacddc8753ad41808da2a57f2f5a986b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eacddc8753ad41808da2a57f2f5a986b" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 44933 } } }
I20260806 02:45:18.056792 2655 sys_catalog.cc:458] T 00000000000000000000000000000000 P eacddc8753ad41808da2a57f2f5a986b [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:18.057063 2657 sys_catalog.cc:455] T 00000000000000000000000000000000 P eacddc8753ad41808da2a57f2f5a986b [sys.catalog]: SysCatalogTable state changed. Reason: New leader eacddc8753ad41808da2a57f2f5a986b. Latest consensus state: current_term: 1 leader_uuid: "eacddc8753ad41808da2a57f2f5a986b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eacddc8753ad41808da2a57f2f5a986b" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 44933 } } }
I20260806 02:45:18.057156 2657 sys_catalog.cc:458] T 00000000000000000000000000000000 P eacddc8753ad41808da2a57f2f5a986b [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:18.057215 2669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:18.057495 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.043389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47880 (local address 127.1.183.190:44933)
0806 02:45:18.043823 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:18.043866 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:18.043905 (+ 39us) server_negotiation.cc:407] Connection header received
0806 02:45:18.044689 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:18.044711 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:18.045078 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:18.045451 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:18.046289 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.047331 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.048279 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.048769 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.048955 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.051889 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:18.051896 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:18.051913 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:18.051968 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:18.053877 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:18.054442 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:18.054448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:18.054454 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:18.054588 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:18.054903 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:18.054906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:18.054908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:18.055458 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:18.055658 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:18.055995 (+ 337us) server_negotiation.cc:299] Negotiation successful
0806 02:45:18.056382 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":131,"threads_started":1}
I20260806 02:45:18.060662 2669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:18.066545 2669 catalog_manager.cc:1347] Generated new cluster ID: e2f17cad04914797b8552b62410cc9c1
I20260806 02:45:18.066637 2669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:18.081481 2669 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:18.082501 2669 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:18.093483 2669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eacddc8753ad41808da2a57f2f5a986b: Generated new TSK 0
I20260806 02:45:18.094442 2669 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
I20260806 02:45:18.131486 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.129:0
--local_ip_for_outbound_sockets=127.1.183.129
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:44933
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:44785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:18.255142 2688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:18.255468 2688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:18.255527 2688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:18.260066 2688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:18.260185 2688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:18.260257 2688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.129
I20260806 02:45:18.265913 2688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:44785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:44933
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.129
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:18.267076 2688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:18.268180 2688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:18.278680 2697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:18.278725 2698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:18.278678 2700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:18.278990 2688 server_base.cc:1034] running on GCE node
I20260806 02:45:18.279537 2688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:18.280263 2688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:18.281472 2688 hybrid_clock.cc:648] HybridClock initialized: now 1785984318281447 us; error 41 us; skew 500 ppm
Aug 06 02:45:18 dist-test-slave-gmjp krb5kdc[2518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984318, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:18.284474 2688 init.cc:377] Logged in from keytab as kudu/127.1.183.129@KRBTEST.COM (short username kudu)
I20260806 02:45:18.285398 2688 webserver.cc:460] Webserver started at http://127.1.183.129:35809/ using document root <none> and password file <none>
I20260806 02:45:18.285969 2688 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:18.286031 2688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:18.286207 2688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:18.288457 2688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-0/data/instance:
uuid: "3550c21b3afa458985aaaa00cee12d00"
format_stamp: "Formatted at 2026-08-06 02:45:18 on dist-test-slave-gmjp"
server_key: "914e4114225629f83390383fcbf76976"
server_key_iv: "e2697c88d66c42ed2df8ab29cca8a54f"
server_key_version: "encryptionkey@0"
I20260806 02:45:18.289105 2688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance:
uuid: "3550c21b3afa458985aaaa00cee12d00"
format_stamp: "Formatted at 2026-08-06 02:45:18 on dist-test-slave-gmjp"
server_key: "914e4114225629f83390383fcbf76976"
server_key_iv: "e2697c88d66c42ed2df8ab29cca8a54f"
server_key_version: "encryptionkey@0"
I20260806 02:45:18.293476 2688 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260806 02:45:18.296382 2711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:18.297871 2688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:45:18.298013 2688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-0/wal
uuid: "3550c21b3afa458985aaaa00cee12d00"
format_stamp: "Formatted at 2026-08-06 02:45:18 on dist-test-slave-gmjp"
server_key: "914e4114225629f83390383fcbf76976"
server_key_iv: "e2697c88d66c42ed2df8ab29cca8a54f"
server_key_version: "encryptionkey@0"
I20260806 02:45:18.298146 2688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:18.309523 2688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:18.313076 2688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:18.313300 2688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:18.313994 2688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:18.314924 2688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:18.314975 2688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:18.315045 2688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:18.315076 2688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:18.324934 2688 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.129:35195
I20260806 02:45:18.324956 2887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.129:35195 every 8 connection(s)
I20260806 02:45:18.325917 2688 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
I20260806 02:45:18.329854 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 2688
I20260806 02:45:18.329975 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance
I20260806 02:45:18.330246 1758 external_mini_cluster.cc:1428] Setting key bb646b3e087c03d219ba1215e1dd435c
Aug 06 02:45:18 dist-test-slave-gmjp krb5kdc[2518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984318, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.183.130@KRBTEST.COM; defaulting to no policy
I20260806 02:45:18.342170 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.327824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:33823 (local address 127.1.183.190:44933)
0806 02:45:18.327985 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:18.327990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:18.328916 (+ 926us) server_negotiation.cc:407] Connection header received
0806 02:45:18.330669 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:18.330673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:18.330745 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:18.330879 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:18.332482 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.333050 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.334583 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.334983 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.335104 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.338082 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:18.338085 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:18.338087 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:18.338112 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:18.339608 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:18.340320 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:18.340323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:18.340324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:18.340424 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:18.340878 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:18.340881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:18.340882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:18.341079 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:18.341157 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:18.341919 (+ 762us) server_negotiation.cc:299] Negotiation successful
0806 02:45:18.342036 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 02:45:18.343323 2890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.328277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:44933 (local address 127.1.183.129:33823)
0806 02:45:18.328744 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:18.328789 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:18.330398 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:18.331130 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:18.331138 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:18.331545 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:18.332306 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:18.332333 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.333188 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.333195 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:18.334453 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:18.334464 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.335155 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.335158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:18.335312 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.336043 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:18.336067 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:18.337880 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:18.339738 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:18.339748 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:18.339768 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:18.340189 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:18.340527 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:18.340533 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:18.340538 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:18.340749 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:18.341191 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:18.341203 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:18.341586 (+ 383us) client_negotiation.cc:769] Sending connection context
0806 02:45:18.341830 (+ 244us) client_negotiation.cc:240] Negotiation successful
0806 02:45:18.342149 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":108,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.130@KRBTEST.COM" created.
I20260806 02:45:18.345110 2888 heartbeater.cc:347] Connected to a master server at 127.1.183.190:44933
I20260806 02:45:18.345376 2888 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:18.346122 2888 heartbeater.cc:511] Master 127.1.183.190:44933 requested a full tablet report, sending...
I20260806 02:45:18.348115 2585 ts_manager.cc:194] Registered new tserver with Master: 3550c21b3afa458985aaaa00cee12d00 (127.1.183.129:35195)
I20260806 02:45:18.349875 2585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.129@KRBTEST.COM'} at 127.1.183.129:33823
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
I20260806 02:45:18.363767 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.130:0
--local_ip_for_outbound_sockets=127.1.183.130
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:44933
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:44785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:18.484047 2893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:18.484386 2893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:18.484434 2893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:18.488622 2893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:18.488698 2893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:18.488760 2893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.130
I20260806 02:45:18.494508 2893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:44785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:44933
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.130
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:18.495769 2893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:18.497009 2893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:18.507843 2899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:18.507826 2901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:18.507954 2903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:18.508358 2893 server_base.cc:1034] running on GCE node
I20260806 02:45:18.508826 2893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:18.509590 2893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:18.510793 2893 hybrid_clock.cc:648] HybridClock initialized: now 1785984318510774 us; error 34 us; skew 500 ppm
Aug 06 02:45:18 dist-test-slave-gmjp krb5kdc[2518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984318, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:18.513896 2893 init.cc:377] Logged in from keytab as kudu/127.1.183.130@KRBTEST.COM (short username kudu)
I20260806 02:45:18.514894 2893 webserver.cc:460] Webserver started at http://127.1.183.130:37789/ using document root <none> and password file <none>
I20260806 02:45:18.515506 2893 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:18.515600 2893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:18.515846 2893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:18.518220 2893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-1/data/instance:
uuid: "a69d818e02714c72bdd687a79e5cb8d3"
format_stamp: "Formatted at 2026-08-06 02:45:18 on dist-test-slave-gmjp"
server_key: "86e3550ff158bde30e54c338a6ebf587"
server_key_iv: "dbb63cac8d5366f09683c1bce430dad4"
server_key_version: "encryptionkey@0"
I20260806 02:45:18.518883 2893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance:
uuid: "a69d818e02714c72bdd687a79e5cb8d3"
format_stamp: "Formatted at 2026-08-06 02:45:18 on dist-test-slave-gmjp"
server_key: "86e3550ff158bde30e54c338a6ebf587"
server_key_iv: "dbb63cac8d5366f09683c1bce430dad4"
server_key_version: "encryptionkey@0"
I20260806 02:45:18.523090 2893 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:45:18.525882 2911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:18.527186 2893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:45:18.527339 2893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-1/wal
uuid: "a69d818e02714c72bdd687a79e5cb8d3"
format_stamp: "Formatted at 2026-08-06 02:45:18 on dist-test-slave-gmjp"
server_key: "86e3550ff158bde30e54c338a6ebf587"
server_key_iv: "dbb63cac8d5366f09683c1bce430dad4"
server_key_version: "encryptionkey@0"
I20260806 02:45:18.527467 2893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:18.541873 2893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:18.545159 2893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:18.545380 2893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:18.546041 2893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:18.546947 2893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:18.547019 2893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:18.547089 2893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:18.547135 2893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:18.556900 2893 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.130:39727
I20260806 02:45:18.556957 3109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.130:39727 every 8 connection(s)
I20260806 02:45:18.558007 2893 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
I20260806 02:45:18.561861 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 2893
I20260806 02:45:18.562052 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance
I20260806 02:45:18.562405 1758 external_mini_cluster.cc:1428] Setting key acc97f25db7297c9247ee9128cc1dfad
Aug 06 02:45:18 dist-test-slave-gmjp krb5kdc[2518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984318, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.183.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.131@KRBTEST.COM" created.
I20260806 02:45:18.577392 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.560232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:52197 (local address 127.1.183.190:44933)
0806 02:45:18.560402 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:18.560408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:18.561460 (+ 1052us) server_negotiation.cc:407] Connection header received
0806 02:45:18.564261 (+ 2801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:18.564266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:18.564341 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:18.564440 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:18.566369 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.567274 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.569170 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.569428 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.569529 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.572965 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:18.572968 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:18.572970 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:18.573000 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:18.574693 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:18.575270 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:18.575273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:18.575275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:18.575373 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:18.575768 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:18.575776 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:18.575777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:18.575984 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:18.576100 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:18.577061 (+ 961us) server_negotiation.cc:299] Negotiation successful
0806 02:45:18.577215 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:45:18.578583 3115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.560706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:44933 (local address 127.1.183.130:52197)
0806 02:45:18.561280 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:18.561333 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:18.563055 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:18.564731 (+ 1676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:18.564745 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:18.565340 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:18.566121 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:18.566145 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.567445 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.567448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:18.568995 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:18.569012 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.569632 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.569640 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:18.569886 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.570588 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:18.570615 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:18.572733 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:18.574834 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:18.574846 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:18.574863 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:18.575149 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:18.575485 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:18.575488 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:18.575490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:18.575661 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:18.576099 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:18.576108 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:18.576682 (+ 574us) client_negotiation.cc:769] Sending connection context
0806 02:45:18.577005 (+ 323us) client_negotiation.cc:240] Negotiation successful
0806 02:45:18.577396 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":157,"threads_started":1}
I20260806 02:45:18.580300 3111 heartbeater.cc:347] Connected to a master server at 127.1.183.190:44933
I20260806 02:45:18.580581 3111 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:18.581139 3111 heartbeater.cc:511] Master 127.1.183.190:44933 requested a full tablet report, sending...
I20260806 02:45:18.582397 2585 ts_manager.cc:194] Registered new tserver with Master: a69d818e02714c72bdd687a79e5cb8d3 (127.1.183.130:39727)
I20260806 02:45:18.583331 2585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.130@KRBTEST.COM'} at 127.1.183.130:52197
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
I20260806 02:45:18.596463 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.131:0
--local_ip_for_outbound_sockets=127.1.183.131
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:44933
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:44785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:18.719416 3118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:18.719739 3118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:18.719791 3118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:18.723922 3118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:18.724010 3118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:18.724080 3118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.131
I20260806 02:45:18.729687 3118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:44785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:44933
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.131
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:18.730837 3118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:18.732028 3118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:18.742623 3130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:18.742619 3124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:18.742623 3125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:18.743163 3118 server_base.cc:1034] running on GCE node
I20260806 02:45:18.743628 3118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:18.744423 3118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:18.745625 3118 hybrid_clock.cc:648] HybridClock initialized: now 1785984318745608 us; error 37 us; skew 500 ppm
Aug 06 02:45:18 dist-test-slave-gmjp krb5kdc[2518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984318, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:18.748723 3118 init.cc:377] Logged in from keytab as kudu/127.1.183.131@KRBTEST.COM (short username kudu)
I20260806 02:45:18.749675 3118 webserver.cc:460] Webserver started at http://127.1.183.131:46301/ using document root <none> and password file <none>
I20260806 02:45:18.750269 3118 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:18.750345 3118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:18.750525 3118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:18.752801 3118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-2/data/instance:
uuid: "f7f0e5c81845416eab1736ab2d8c659b"
format_stamp: "Formatted at 2026-08-06 02:45:18 on dist-test-slave-gmjp"
server_key: "2e6e6010294d3933114053eebf345ba2"
server_key_iv: "8b8c5bbf737872208f4814f98a005975"
server_key_version: "encryptionkey@0"
I20260806 02:45:18.753355 3118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance:
uuid: "f7f0e5c81845416eab1736ab2d8c659b"
format_stamp: "Formatted at 2026-08-06 02:45:18 on dist-test-slave-gmjp"
server_key: "2e6e6010294d3933114053eebf345ba2"
server_key_iv: "8b8c5bbf737872208f4814f98a005975"
server_key_version: "encryptionkey@0"
I20260806 02:45:18.757701 3118 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:45:18.760386 3140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:18.761724 3118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:18.761888 3118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-2/wal
uuid: "f7f0e5c81845416eab1736ab2d8c659b"
format_stamp: "Formatted at 2026-08-06 02:45:18 on dist-test-slave-gmjp"
server_key: "2e6e6010294d3933114053eebf345ba2"
server_key_iv: "8b8c5bbf737872208f4814f98a005975"
server_key_version: "encryptionkey@0"
I20260806 02:45:18.762022 3118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:18.783668 3118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:18.787153 3118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:18.787407 3118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:18.788092 3118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:18.789047 3118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:18.789119 3118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:18.789194 3118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:18.789235 3118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:18.798569 3118 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.131:45037
I20260806 02:45:18.798593 3301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.131:45037 every 8 connection(s)
I20260806 02:45:18.799548 3118 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
I20260806 02:45:18.804793 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 3118
I20260806 02:45:18.804962 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance
I20260806 02:45:18.805233 1758 external_mini_cluster.cc:1428] Setting key 04444a3a036713193b6a79c4951e7188
Aug 06 02:45:18 dist-test-slave-gmjp krb5kdc[2518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984318, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:18.814854 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.801864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:57739 (local address 127.1.183.190:44933)
0806 02:45:18.802028 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:18.802031 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:18.802931 (+ 900us) server_negotiation.cc:407] Connection header received
0806 02:45:18.804449 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:18.804453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:18.804513 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:18.804617 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:18.806104 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.806810 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.808087 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.808382 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.808490 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.811080 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:18.811083 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:18.811086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:18.811122 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:18.812658 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:18.813210 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:18.813214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:18.813216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:18.813317 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:18.813695 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:18.813698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:18.813700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:18.813903 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:18.814012 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:18.814538 (+ 526us) server_negotiation.cc:299] Negotiation successful
0806 02:45:18.814677 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:45:18.815742 3305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.802245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:44933 (local address 127.1.183.131:57739)
0806 02:45:18.802779 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:18.802825 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:18.804217 (+ 1392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:18.804811 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:18.804818 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:18.805285 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:18.805956 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:18.805975 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.806939 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.806942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:18.807961 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:18.807971 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.808539 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.808543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:18.808725 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.809312 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:18.809329 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:18.810897 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:18.812797 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:18.812805 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:18.812826 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:18.813101 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:18.813413 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:18.813416 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:18.813418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:18.813565 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:18.814012 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:18.814020 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:18.814296 (+ 276us) client_negotiation.cc:769] Sending connection context
0806 02:45:18.814497 (+ 201us) client_negotiation.cc:240] Negotiation successful
0806 02:45:18.814875 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":150,"threads_started":1}
I20260806 02:45:18.817075 3303 heartbeater.cc:347] Connected to a master server at 127.1.183.190:44933
I20260806 02:45:18.817348 3303 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:18.817970 3303 heartbeater.cc:511] Master 127.1.183.190:44933 requested a full tablet report, sending...
I20260806 02:45:18.819283 2585 ts_manager.cc:194] Registered new tserver with Master: f7f0e5c81845416eab1736ab2d8c659b (127.1.183.131:45037)
I20260806 02:45:18.819751 1758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:45:18.820097 2585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.131@KRBTEST.COM'} at 127.1.183.131:57739
I20260806 02:45:18.830991 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.823546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47888 (local address 127.1.183.190:44933)
0806 02:45:18.823675 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:18.823680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:18.823838 (+ 158us) server_negotiation.cc:407] Connection header received
0806 02:45:18.823984 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:18.823987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:18.824034 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:18.824117 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:18.824911 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.825439 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.826277 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.826503 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.826588 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.827983 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:18.827986 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:18.827987 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:18.828012 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:18.829436 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:18.829835 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:18.829837 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:18.829839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:18.829947 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:18.830175 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:18.830177 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:18.830178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:18.830450 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:18.830555 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:18.830762 (+ 207us) server_negotiation.cc:299] Negotiation successful
0806 02:45:18.830865 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:45:18.836099 2585 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:45:18.839241 2585 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:45:18.854720 3323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.848118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:35195 (local address 127.0.0.1:54848)
0806 02:45:18.849094 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:18.849149 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:18.849278 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:18.850042 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:18.850048 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:18.850071 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:18.850215 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:18.850225 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.851683 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.851686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:18.853249 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:18.853262 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.854109 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.854113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:18.854233 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.854272 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 02:45:18.854429 (+ 157us) client_negotiation.cc:240] Negotiation successful
0806 02:45:18.854565 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"mutex_wait_us":39,"thread_start_us":101,"threads_started":1}
I20260806 02:45:18.854732 3325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.849014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54848 (local address 127.1.183.129:35195)
0806 02:45:18.849542 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:18.849551 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:18.849576 (+ 25us) server_negotiation.cc:407] Connection header received
0806 02:45:18.849657 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:18.849664 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:18.849892 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:18.850052 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:18.850349 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.851565 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.853416 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.853978 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.854109 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.854277 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:18.854466 (+ 189us) server_negotiation.cc:299] Negotiation successful
0806 02:45:18.854569 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":70,"threads_started":1}
I20260806 02:45:18.854925 3319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.848121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:45037 (local address 127.0.0.1:47382)
0806 02:45:18.849422 (+ 1301us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:18.849438 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:18.849531 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:18.850208 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:18.850212 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:18.850243 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:18.850462 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:18.850467 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.851871 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.851874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:18.853256 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:18.853262 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.854512 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.854516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:18.854606 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.854628 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:45:18.854697 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 02:45:18.854767 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1209,"mutex_wait_us":195,"spinlock_wait_cycles":12928,"thread_start_us":116,"threads_started":1}
I20260806 02:45:18.855341 3322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.848281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47382 (local address 127.1.183.131:45037)
0806 02:45:18.849272 (+ 991us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:18.849280 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:18.849630 (+ 350us) server_negotiation.cc:407] Connection header received
0806 02:45:18.849712 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:18.849718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:18.849936 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:18.850089 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:18.850584 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.851687 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.853386 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.854201 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.854300 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.854449 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:18.855029 (+ 580us) server_negotiation.cc:299] Negotiation successful
0806 02:45:18.855131 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":898,"thread_start_us":126,"threads_started":1}
I20260806 02:45:18.856607 3318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.848482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:39727 (local address 127.0.0.1:37272)
0806 02:45:18.849502 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:18.849515 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:18.849603 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:18.850208 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:18.850212 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:18.850242 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:18.850461 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:18.850467 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.852976 (+ 2509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.852982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:18.854146 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:18.854160 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.856103 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.856107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:18.856268 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.856294 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 02:45:18.856360 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 02:45:18.856440 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":959,"mutex_wait_us":101,"spinlock_wait_cycles":5760,"thread_start_us":97,"threads_started":1}
I20260806 02:45:18.857205 3324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.848789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37272 (local address 127.1.183.130:39727)
0806 02:45:18.849158 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:18.849171 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:18.849696 (+ 525us) server_negotiation.cc:407] Connection header received
0806 02:45:18.849769 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:18.849773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:18.849926 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:18.850057 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:18.851748 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.852833 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.854303 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.855125 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.855271 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.856881 (+ 1610us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:18.856957 (+ 76us) server_negotiation.cc:299] Negotiation successful
0806 02:45:18.857052 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":181,"threads_started":1}
I20260806 02:45:18.858273 2794 tablet_service.cc:1467] Processing CreateTablet for tablet 088de788058d472da3cebda6380327df (DEFAULT_TABLE table=test-table [id=46407530c92649df90348d221d054b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:18.858270 3209 tablet_service.cc:1467] Processing CreateTablet for tablet 088de788058d472da3cebda6380327df (DEFAULT_TABLE table=test-table [id=46407530c92649df90348d221d054b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:18.859283 2794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 088de788058d472da3cebda6380327df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:18.859308 3209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 088de788058d472da3cebda6380327df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:18.865785 3326 tablet_bootstrap.cc:492] T 088de788058d472da3cebda6380327df P 3550c21b3afa458985aaaa00cee12d00: Bootstrap starting.
I20260806 02:45:18.867475 3327 tablet_bootstrap.cc:492] T 088de788058d472da3cebda6380327df P f7f0e5c81845416eab1736ab2d8c659b: Bootstrap starting.
I20260806 02:45:18.867383 2995 tablet_service.cc:1467] Processing CreateTablet for tablet 088de788058d472da3cebda6380327df (DEFAULT_TABLE table=test-table [id=46407530c92649df90348d221d054b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:18.868494 3326 tablet_bootstrap.cc:654] T 088de788058d472da3cebda6380327df P 3550c21b3afa458985aaaa00cee12d00: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:18.868619 2995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 088de788058d472da3cebda6380327df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:18.869434 3326 log.cc:824] T 088de788058d472da3cebda6380327df P 3550c21b3afa458985aaaa00cee12d00: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:18.870106 3327 tablet_bootstrap.cc:654] T 088de788058d472da3cebda6380327df P f7f0e5c81845416eab1736ab2d8c659b: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:18.871075 3327 log.cc:824] T 088de788058d472da3cebda6380327df P f7f0e5c81845416eab1736ab2d8c659b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:18.871861 3326 tablet_bootstrap.cc:492] T 088de788058d472da3cebda6380327df P 3550c21b3afa458985aaaa00cee12d00: No bootstrap required, opened a new log
I20260806 02:45:18.872102 3326 ts_tablet_manager.cc:1397] T 088de788058d472da3cebda6380327df P 3550c21b3afa458985aaaa00cee12d00: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 02:45:18.873294 3327 tablet_bootstrap.cc:492] T 088de788058d472da3cebda6380327df P f7f0e5c81845416eab1736ab2d8c659b: No bootstrap required, opened a new log
I20260806 02:45:18.873505 3327 ts_tablet_manager.cc:1397] T 088de788058d472da3cebda6380327df P f7f0e5c81845416eab1736ab2d8c659b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:45:18.875665 3331 tablet_bootstrap.cc:492] T 088de788058d472da3cebda6380327df P a69d818e02714c72bdd687a79e5cb8d3: Bootstrap starting.
I20260806 02:45:18.876967 3326 raft_consensus.cc:348] T 088de788058d472da3cebda6380327df P 3550c21b3afa458985aaaa00cee12d00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7f0e5c81845416eab1736ab2d8c659b" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45037 } } peers { permanent_uuid: "3550c21b3afa458985aaaa00cee12d00" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 35195 } } peers { permanent_uuid: "a69d818e02714c72bdd687a79e5cb8d3" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39727 } }
I20260806 02:45:18.877269 3326 raft_consensus.cc:374] T 088de788058d472da3cebda6380327df P 3550c21b3afa458985aaaa00cee12d00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:18.877338 3326 raft_consensus.cc:729] T 088de788058d472da3cebda6380327df P 3550c21b3afa458985aaaa00cee12d00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3550c21b3afa458985aaaa00cee12d00, State: Initialized, Role: FOLLOWER
I20260806 02:45:18.877851 3326 consensus_queue.cc:260] T 088de788058d472da3cebda6380327df P 3550c21b3afa458985aaaa00cee12d00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7f0e5c81845416eab1736ab2d8c659b" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45037 } } peers { permanent_uuid: "3550c21b3afa458985aaaa00cee12d00" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 35195 } } peers { permanent_uuid: "a69d818e02714c72bdd687a79e5cb8d3" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39727 } }
I20260806 02:45:18.877872 3327 raft_consensus.cc:348] T 088de788058d472da3cebda6380327df P f7f0e5c81845416eab1736ab2d8c659b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7f0e5c81845416eab1736ab2d8c659b" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45037 } } peers { permanent_uuid: "3550c21b3afa458985aaaa00cee12d00" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 35195 } } peers { permanent_uuid: "a69d818e02714c72bdd687a79e5cb8d3" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39727 } }
I20260806 02:45:18.878139 3327 raft_consensus.cc:374] T 088de788058d472da3cebda6380327df P f7f0e5c81845416eab1736ab2d8c659b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:18.878221 3327 raft_consensus.cc:729] T 088de788058d472da3cebda6380327df P f7f0e5c81845416eab1736ab2d8c659b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7f0e5c81845416eab1736ab2d8c659b, State: Initialized, Role: FOLLOWER
I20260806 02:45:18.878657 2888 heartbeater.cc:503] Master 127.1.183.190:44933 was elected leader, sending a full tablet report...
I20260806 02:45:18.878696 3327 consensus_queue.cc:260] T 088de788058d472da3cebda6380327df P f7f0e5c81845416eab1736ab2d8c659b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7f0e5c81845416eab1736ab2d8c659b" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45037 } } peers { permanent_uuid: "3550c21b3afa458985aaaa00cee12d00" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 35195 } } peers { permanent_uuid: "a69d818e02714c72bdd687a79e5cb8d3" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39727 } }
I20260806 02:45:18.879297 3331 tablet_bootstrap.cc:654] T 088de788058d472da3cebda6380327df P a69d818e02714c72bdd687a79e5cb8d3: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:18.879447 3303 heartbeater.cc:503] Master 127.1.183.190:44933 was elected leader, sending a full tablet report...
I20260806 02:45:18.879760 3327 ts_tablet_manager.cc:1428] T 088de788058d472da3cebda6380327df P f7f0e5c81845416eab1736ab2d8c659b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 02:45:18.880347 3326 ts_tablet_manager.cc:1428] T 088de788058d472da3cebda6380327df P 3550c21b3afa458985aaaa00cee12d00: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20260806 02:45:18.880904 3331 log.cc:824] T 088de788058d472da3cebda6380327df P a69d818e02714c72bdd687a79e5cb8d3: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:18.883123 3331 tablet_bootstrap.cc:492] T 088de788058d472da3cebda6380327df P a69d818e02714c72bdd687a79e5cb8d3: No bootstrap required, opened a new log
I20260806 02:45:18.883359 3331 ts_tablet_manager.cc:1397] T 088de788058d472da3cebda6380327df P a69d818e02714c72bdd687a79e5cb8d3: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260806 02:45:18.886760 3331 raft_consensus.cc:348] T 088de788058d472da3cebda6380327df P a69d818e02714c72bdd687a79e5cb8d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7f0e5c81845416eab1736ab2d8c659b" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45037 } } peers { permanent_uuid: "3550c21b3afa458985aaaa00cee12d00" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 35195 } } peers { permanent_uuid: "a69d818e02714c72bdd687a79e5cb8d3" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39727 } }
I20260806 02:45:18.887012 3331 raft_consensus.cc:374] T 088de788058d472da3cebda6380327df P a69d818e02714c72bdd687a79e5cb8d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:18.887089 3331 raft_consensus.cc:729] T 088de788058d472da3cebda6380327df P a69d818e02714c72bdd687a79e5cb8d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69d818e02714c72bdd687a79e5cb8d3, State: Initialized, Role: FOLLOWER
I20260806 02:45:18.887506 3331 consensus_queue.cc:260] T 088de788058d472da3cebda6380327df P a69d818e02714c72bdd687a79e5cb8d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7f0e5c81845416eab1736ab2d8c659b" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45037 } } peers { permanent_uuid: "3550c21b3afa458985aaaa00cee12d00" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 35195 } } peers { permanent_uuid: "a69d818e02714c72bdd687a79e5cb8d3" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39727 } }
I20260806 02:45:18.888226 3111 heartbeater.cc:503] Master 127.1.183.190:44933 was elected leader, sending a full tablet report...
I20260806 02:45:18.888541 3331 ts_tablet_manager.cc:1428] T 088de788058d472da3cebda6380327df P a69d818e02714c72bdd687a79e5cb8d3: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260806 02:45:18.984767 3336 raft_consensus.cc:482] T 088de788058d472da3cebda6380327df P f7f0e5c81845416eab1736ab2d8c659b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:45:18.985041 3336 raft_consensus.cc:504] T 088de788058d472da3cebda6380327df P f7f0e5c81845416eab1736ab2d8c659b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7f0e5c81845416eab1736ab2d8c659b" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45037 } } peers { permanent_uuid: "3550c21b3afa458985aaaa00cee12d00" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 35195 } } peers { permanent_uuid: "a69d818e02714c72bdd687a79e5cb8d3" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39727 } }
I20260806 02:45:18.986335 3336 leader_election.cc:288] T 088de788058d472da3cebda6380327df P f7f0e5c81845416eab1736ab2d8c659b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3550c21b3afa458985aaaa00cee12d00 (127.1.183.129:35195), a69d818e02714c72bdd687a79e5cb8d3 (127.1.183.130:39727)
I20260806 02:45:18.991557 3324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.986898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:33075 (local address 127.1.183.130:39727)
0806 02:45:18.987046 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:18.987051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:18.987314 (+ 263us) server_negotiation.cc:407] Connection header received
0806 02:45:18.987445 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:18.987447 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:18.987491 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:18.987582 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:18.987822 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.988569 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.990170 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.990803 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.990943 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.990999 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:18.991336 (+ 337us) server_negotiation.cc:299] Negotiation successful
0806 02:45:18.991417 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 02:45:18.991557 3325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.986784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:60557 (local address 127.1.183.129:35195)
0806 02:45:18.986940 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:18.986948 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:18.986971 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:45:18.987136 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:18.987139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:18.987186 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:18.987246 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:18.987600 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.988443 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.990170 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.990809 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.990942 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.990999 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:18.991336 (+ 337us) server_negotiation.cc:299] Negotiation successful
0806 02:45:18.991417 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:45:18.991619 3305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.986591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:35195 (local address 127.1.183.131:60557)
0806 02:45:18.986833 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:18.986849 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:18.986956 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:18.987300 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:18.987303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:18.987342 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:18.987486 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:18.987492 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.988561 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.988563 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:18.990035 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:18.990047 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.990945 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.990949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:18.991167 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.991229 (+ 62us) client_negotiation.cc:769] Sending connection context
0806 02:45:18.991308 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 02:45:18.991450 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":191,"spinlock_wait_cycles":31872}
I20260806 02:45:18.991828 3339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:18.986769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:39727 (local address 127.1.183.131:33075)
0806 02:45:18.987181 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:18.987193 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:18.987340 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:18.987597 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:18.987601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:18.987624 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:18.987717 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:18.987721 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.988665 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.988667 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:18.990035 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:18.990047 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:18.990945 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:18.990949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:18.991140 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:18.991225 (+ 85us) client_negotiation.cc:769] Sending connection context
0806 02:45:18.991316 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 02:45:18.991450 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":170,"spinlock_wait_cycles":768,"thread_start_us":153,"threads_started":1}
I20260806 02:45:18.992426 2821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088de788058d472da3cebda6380327df" candidate_uuid: "f7f0e5c81845416eab1736ab2d8c659b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3550c21b3afa458985aaaa00cee12d00" is_pre_election: true
I20260806 02:45:18.992545 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088de788058d472da3cebda6380327df" candidate_uuid: "f7f0e5c81845416eab1736ab2d8c659b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69d818e02714c72bdd687a79e5cb8d3" is_pre_election: true
I20260806 02:45:18.992794 3024 raft_consensus.cc:2454] T 088de788058d472da3cebda6380327df P a69d818e02714c72bdd687a79e5cb8d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7f0e5c81845416eab1736ab2d8c659b in term 0.
I20260806 02:45:18.992794 2821 raft_consensus.cc:2454] T 088de788058d472da3cebda6380327df P 3550c21b3afa458985aaaa00cee12d00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7f0e5c81845416eab1736ab2d8c659b in term 0.
I20260806 02:45:18.993345 3141 leader_election.cc:302] T 088de788058d472da3cebda6380327df P f7f0e5c81845416eab1736ab2d8c659b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3550c21b3afa458985aaaa00cee12d00, f7f0e5c81845416eab1736ab2d8c659b; no voters:
I20260806 02:45:18.993619 3336 raft_consensus.cc:2781] T 088de788058d472da3cebda6380327df P f7f0e5c81845416eab1736ab2d8c659b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:45:18.993712 3336 raft_consensus.cc:482] T 088de788058d472da3cebda6380327df P f7f0e5c81845416eab1736ab2d8c659b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:45:18.993762 3336 raft_consensus.cc:3037] T 088de788058d472da3cebda6380327df P f7f0e5c81845416eab1736ab2d8c659b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:18.994887 3336 raft_consensus.cc:504] T 088de788058d472da3cebda6380327df P f7f0e5c81845416eab1736ab2d8c659b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7f0e5c81845416eab1736ab2d8c659b" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45037 } } peers { permanent_uuid: "3550c21b3afa458985aaaa00cee12d00" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 35195 } } peers { permanent_uuid: "a69d818e02714c72bdd687a79e5cb8d3" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39727 } }
I20260806 02:45:18.995254 3336 leader_election.cc:288] T 088de788058d472da3cebda6380327df P f7f0e5c81845416eab1736ab2d8c659b [CANDIDATE]: Term 1 election: Requested vote from peers 3550c21b3afa458985aaaa00cee12d00 (127.1.183.129:35195), a69d818e02714c72bdd687a79e5cb8d3 (127.1.183.130:39727)
I20260806 02:45:18.995635 2821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088de788058d472da3cebda6380327df" candidate_uuid: "f7f0e5c81845416eab1736ab2d8c659b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3550c21b3afa458985aaaa00cee12d00"
I20260806 02:45:18.995703 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "088de788058d472da3cebda6380327df" candidate_uuid: "f7f0e5c81845416eab1736ab2d8c659b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69d818e02714c72bdd687a79e5cb8d3"
I20260806 02:45:18.995779 2821 raft_consensus.cc:3037] T 088de788058d472da3cebda6380327df P 3550c21b3afa458985aaaa00cee12d00 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:18.995821 3024 raft_consensus.cc:3037] T 088de788058d472da3cebda6380327df P a69d818e02714c72bdd687a79e5cb8d3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:18.997221 2821 raft_consensus.cc:2454] T 088de788058d472da3cebda6380327df P 3550c21b3afa458985aaaa00cee12d00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7f0e5c81845416eab1736ab2d8c659b in term 1.
I20260806 02:45:18.997221 3024 raft_consensus.cc:2454] T 088de788058d472da3cebda6380327df P a69d818e02714c72bdd687a79e5cb8d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7f0e5c81845416eab1736ab2d8c659b in term 1.
I20260806 02:45:18.997560 3141 leader_election.cc:302] T 088de788058d472da3cebda6380327df P f7f0e5c81845416eab1736ab2d8c659b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3550c21b3afa458985aaaa00cee12d00, f7f0e5c81845416eab1736ab2d8c659b; no voters:
I20260806 02:45:18.997749 3336 raft_consensus.cc:2781] T 088de788058d472da3cebda6380327df P f7f0e5c81845416eab1736ab2d8c659b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:18.998010 3336 raft_consensus.cc:686] T 088de788058d472da3cebda6380327df P f7f0e5c81845416eab1736ab2d8c659b [term 1 LEADER]: Becoming Leader. State: Replica: f7f0e5c81845416eab1736ab2d8c659b, State: Running, Role: LEADER
I20260806 02:45:18.998343 3336 consensus_queue.cc:237] T 088de788058d472da3cebda6380327df P f7f0e5c81845416eab1736ab2d8c659b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7f0e5c81845416eab1736ab2d8c659b" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45037 } } peers { permanent_uuid: "3550c21b3afa458985aaaa00cee12d00" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 35195 } } peers { permanent_uuid: "a69d818e02714c72bdd687a79e5cb8d3" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39727 } }
I20260806 02:45:19.002136 2585 catalog_manager.cc:5495] T 088de788058d472da3cebda6380327df P f7f0e5c81845416eab1736ab2d8c659b reported cstate change: term changed from 0 to 1, leader changed from <none> to f7f0e5c81845416eab1736ab2d8c659b (127.1.183.131). New cstate: current_term: 1 leader_uuid: "f7f0e5c81845416eab1736ab2d8c659b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7f0e5c81845416eab1736ab2d8c659b" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3550c21b3afa458985aaaa00cee12d00" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 35195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a69d818e02714c72bdd687a79e5cb8d3" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39727 } health_report { overall_health: UNKNOWN } } }
W20260806 02:45:19.050834 3304 tablet.cc:2369] T 088de788058d472da3cebda6380327df P f7f0e5c81845416eab1736ab2d8c659b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 02:45:19 dist-test-slave-gmjp krb5kdc[2518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.129@KRBTEST.COM
I20260806 02:45:19.056799 3325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:19.038094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54860 (local address 127.1.183.129:35195)
0806 02:45:19.038277 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:19.038282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:19.038395 (+ 113us) server_negotiation.cc:407] Connection header received
0806 02:45:19.038572 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:19.038575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:19.038624 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:19.038715 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:19.039488 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:19.040005 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:19.040920 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:19.041159 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:19.041269 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:19.053058 (+ 11789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:19.053075 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:19.053091 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:19.053128 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:19.054738 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:19.055266 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:19.055273 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:19.055274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:19.055389 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:19.055709 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:19.055713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:19.055715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:19.056053 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:19.056330 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:19.056486 (+ 156us) server_negotiation.cc:299] Negotiation successful
0806 02:45:19.056625 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260806 02:45:19.059407 3113 tablet.cc:2369] T 088de788058d472da3cebda6380327df P a69d818e02714c72bdd687a79e5cb8d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 02:45:19 dist-test-slave-gmjp krb5kdc[2518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984319, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260806 02:45:19.077443 2889 tablet.cc:2369] T 088de788058d472da3cebda6380327df P 3550c21b3afa458985aaaa00cee12d00: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 02:45:19 dist-test-slave-gmjp krb5kdc[2518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984319, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.183.129@KRBTEST.COM
I20260806 02:45:19.086843 3325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:19.077604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54874 (local address 127.1.183.129:35195)
0806 02:45:19.077758 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:19.077763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:19.078012 (+ 249us) server_negotiation.cc:407] Connection header received
0806 02:45:19.078178 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:19.078181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:19.078231 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:19.078319 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:19.079057 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:19.079599 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:19.080538 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:19.080774 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:19.080886 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:19.083543 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:19.083547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:19.083549 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:19.083575 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:19.085098 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:19.085514 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:19.085519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:19.085520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:19.085602 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:19.085917 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:19.085920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:19.085922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:19.086099 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:19.086239 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:19.086534 (+ 295us) server_negotiation.cc:299] Negotiation successful
0806 02:45:19.086670 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Aug 06 02:45:19 dist-test-slave-gmjp krb5kdc[2518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:45:19 dist-test-slave-gmjp krb5kdc[2518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.129@KRBTEST.COM
I20260806 02:45:19.115751 3325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:19.106998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54882 (local address 127.1.183.129:35195)
0806 02:45:19.107142 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:19.107147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:19.107323 (+ 176us) server_negotiation.cc:407] Connection header received
0806 02:45:19.107518 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:19.107521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:19.107569 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:19.107653 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:19.108438 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:19.108935 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:19.109774 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:19.109996 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:19.110097 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:19.112730 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:19.112734 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:19.112736 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:19.112763 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:19.114072 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:19.114509 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:19.114513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:19.114515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:19.114601 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:19.114891 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:19.114895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:19.114896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:19.115075 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:19.115209 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:19.115449 (+ 240us) server_negotiation.cc:299] Negotiation successful
0806 02:45:19.115576 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260806 02:45:19.117483 2760 tablet_service.cc:2908] Rejecting scan request for tablet 088de788058d472da3cebda6380327df: Uninitialized: clean time has not yet been initialized
I20260806 02:45:19.119726 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 2688
I20260806 02:45:19.137473 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 2893
I20260806 02:45:19.150251 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 3118
I20260806 02:45:19.163446 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 2537
2026-08-06T02:45:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2700 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:45:19 dist-test-slave-gmjp krb5kdc[3359](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:45:19 dist-test-slave-gmjp krb5kdc[3359](info): set up 2 sockets
Aug 06 02:45:19 dist-test-slave-gmjp krb5kdc[3359](info): commencing operation
krb5kdc: starting...
W20260806 02:45:20.419575 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.222s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:45:20 dist-test-slave-gmjp krb5kdc[3359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:45:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
I20260806 02:45:20.556360 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:33023
--webserver_interface=127.1.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:41929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:33023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:20.674917 3382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:20.675261 3382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:20.675343 3382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:20.679498 3382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:20.679605 3382 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 02:45:20.679637 3382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:20.679679 3382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:20.679736 3382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:20.679773 3382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:20.685552 3382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:41929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:33023
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:33023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:20.686748 3382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:20.687883 3382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:20.697263 3389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:20.697341 3393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:20.697268 3391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:20.697721 3382 server_base.cc:1034] running on GCE node
I20260806 02:45:20.698330 3382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:20.699517 3382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:20.700681 3382 hybrid_clock.cc:648] HybridClock initialized: now 1785984320700667 us; error 32 us; skew 500 ppm
Aug 06 02:45:20 dist-test-slave-gmjp krb5kdc[3359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984320, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:20.703594 3382 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:45:20.704706 3382 webserver.cc:460] Webserver started at http://127.1.183.190:41729/ using document root <none> and password file <none>
I20260806 02:45:20.705312 3382 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:20.705402 3382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:20.705644 3382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:20.707842 3382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "1154321aa1f74423a374e00d8cef960c"
format_stamp: "Formatted at 2026-08-06 02:45:20 on dist-test-slave-gmjp"
server_key: "dd371af4d730d4b621933dccd3193e77"
server_key_iv: "6c4d2644bf66a53ee1af1058f985c685"
server_key_version: "encryptionkey@0"
I20260806 02:45:20.708511 3382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "1154321aa1f74423a374e00d8cef960c"
format_stamp: "Formatted at 2026-08-06 02:45:20 on dist-test-slave-gmjp"
server_key: "dd371af4d730d4b621933dccd3193e77"
server_key_iv: "6c4d2644bf66a53ee1af1058f985c685"
server_key_version: "encryptionkey@0"
I20260806 02:45:20.712550 3382 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:45:20.715200 3404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:20.716763 3382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:20.716917 3382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "1154321aa1f74423a374e00d8cef960c"
format_stamp: "Formatted at 2026-08-06 02:45:20 on dist-test-slave-gmjp"
server_key: "dd371af4d730d4b621933dccd3193e77"
server_key_iv: "6c4d2644bf66a53ee1af1058f985c685"
server_key_version: "encryptionkey@0"
I20260806 02:45:20.717044 3382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:20.731771 3382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:20.735363 3382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:20.735601 3382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:20.744831 3382 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:33023
I20260806 02:45:20.744851 3495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:33023 every 8 connection(s)
I20260806 02:45:20.745893 3382 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:20.748883 3496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:20.754439 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 3382
I20260806 02:45:20.754557 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:45:20.754878 1758 external_mini_cluster.cc:1428] Setting key f71d30defd1afe9c0bb917e6f933145d
I20260806 02:45:20.755038 3496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1154321aa1f74423a374e00d8cef960c: Bootstrap starting.
I20260806 02:45:20.757771 3496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1154321aa1f74423a374e00d8cef960c: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:20.758684 3496 log.cc:824] T 00000000000000000000000000000000 P 1154321aa1f74423a374e00d8cef960c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:20.760933 3496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1154321aa1f74423a374e00d8cef960c: No bootstrap required, opened a new log
Aug 06 02:45:20 dist-test-slave-gmjp krb5kdc[3359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:20.765092 3496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1154321aa1f74423a374e00d8cef960c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1154321aa1f74423a374e00d8cef960c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33023 } }
I20260806 02:45:20.765455 3496 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1154321aa1f74423a374e00d8cef960c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:20.765534 3496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1154321aa1f74423a374e00d8cef960c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1154321aa1f74423a374e00d8cef960c, State: Initialized, Role: FOLLOWER
I20260806 02:45:20.766096 3496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1154321aa1f74423a374e00d8cef960c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1154321aa1f74423a374e00d8cef960c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33023 } }
I20260806 02:45:20.766280 3496 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1154321aa1f74423a374e00d8cef960c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:20.766364 3496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1154321aa1f74423a374e00d8cef960c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:20.766487 3496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1154321aa1f74423a374e00d8cef960c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:20.767550 3496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1154321aa1f74423a374e00d8cef960c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1154321aa1f74423a374e00d8cef960c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33023 } }
I20260806 02:45:20.767922 3496 leader_election.cc:302] T 00000000000000000000000000000000 P 1154321aa1f74423a374e00d8cef960c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1154321aa1f74423a374e00d8cef960c; no voters:
I20260806 02:45:20.768332 3496 leader_election.cc:288] T 00000000000000000000000000000000 P 1154321aa1f74423a374e00d8cef960c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:20.768448 3501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1154321aa1f74423a374e00d8cef960c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:20.768673 3501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1154321aa1f74423a374e00d8cef960c [term 1 LEADER]: Becoming Leader. State: Replica: 1154321aa1f74423a374e00d8cef960c, State: Running, Role: LEADER
I20260806 02:45:20.769030 3501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1154321aa1f74423a374e00d8cef960c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1154321aa1f74423a374e00d8cef960c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33023 } }
I20260806 02:45:20.769831 3496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1154321aa1f74423a374e00d8cef960c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:20.771044 3503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1154321aa1f74423a374e00d8cef960c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1154321aa1f74423a374e00d8cef960c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1154321aa1f74423a374e00d8cef960c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33023 } } }
I20260806 02:45:20.771178 3503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1154321aa1f74423a374e00d8cef960c [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:20.771221 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:20.756482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38752 (local address 127.1.183.190:33023)
0806 02:45:20.757006 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:20.757025 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:20.757067 (+ 42us) server_negotiation.cc:407] Connection header received
0806 02:45:20.757923 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:20.757953 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:20.758363 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:20.758763 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:20.759797 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:20.760899 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:20.761833 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:20.762383 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:20.762578 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:20.765661 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:20.765668 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:20.765689 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:20.765729 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:20.767651 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:20.768177 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:20.768188 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:20.768198 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:20.768377 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:20.768755 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:20.768758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:20.768760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:20.769279 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:20.769499 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:20.769791 (+ 292us) server_negotiation.cc:299] Negotiation successful
0806 02:45:20.770115 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":150,"threads_started":1}
I20260806 02:45:20.771637 3514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:20.771025 3504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1154321aa1f74423a374e00d8cef960c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1154321aa1f74423a374e00d8cef960c. Latest consensus state: current_term: 1 leader_uuid: "1154321aa1f74423a374e00d8cef960c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1154321aa1f74423a374e00d8cef960c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33023 } } }
I20260806 02:45:20.772601 3504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1154321aa1f74423a374e00d8cef960c [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:20.775166 3514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:20.781383 3514 catalog_manager.cc:1347] Generated new cluster ID: 8ec49627d6184dda947b2a512a833551
I20260806 02:45:20.781481 3514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:20.822907 3514 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:20.824208 3514 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:20.834161 3514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1154321aa1f74423a374e00d8cef960c: Generated new TSK 0
I20260806 02:45:20.834996 3514 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
I20260806 02:45:20.880173 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.129:0
--local_ip_for_outbound_sockets=127.1.183.129
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:33023
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:41929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:20.999636 3530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:20.999974 3530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:21.000067 3530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:21.004228 3530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:21.004333 3530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:21.004411 3530 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:45:21.004490 3530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.129
I20260806 02:45:21.010030 3530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:41929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.183.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.183.190:33023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.129
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:21.011200 3530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:21.012300 3530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:21.022514 3536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:21.022521 3541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:21.022535 3538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:21.022967 3530 server_base.cc:1034] running on GCE node
I20260806 02:45:21.023423 3530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:21.024199 3530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:21.025395 3530 hybrid_clock.cc:648] HybridClock initialized: now 1785984321025369 us; error 36 us; skew 500 ppm
Aug 06 02:45:21 dist-test-slave-gmjp krb5kdc[3359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984321, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:21.028396 3530 init.cc:377] Logged in from keytab as kudu/127.1.183.129@KRBTEST.COM (short username kudu)
I20260806 02:45:21.029320 3530 webserver.cc:460] Webserver started at http://127.1.183.129:38277/ using document root <none> and password file <none>
I20260806 02:45:21.029901 3530 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:21.029990 3530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:21.030216 3530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:21.032425 3530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-0/data/instance:
uuid: "ee07153e366241aa941ec4729842b4f4"
format_stamp: "Formatted at 2026-08-06 02:45:21 on dist-test-slave-gmjp"
server_key: "d98c29d6c3dc30365faf871c5ae5cb34"
server_key_iv: "5899c734865a7e171f95f403e5818f20"
server_key_version: "encryptionkey@0"
I20260806 02:45:21.032996 3530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance:
uuid: "ee07153e366241aa941ec4729842b4f4"
format_stamp: "Formatted at 2026-08-06 02:45:21 on dist-test-slave-gmjp"
server_key: "d98c29d6c3dc30365faf871c5ae5cb34"
server_key_iv: "5899c734865a7e171f95f403e5818f20"
server_key_version: "encryptionkey@0"
I20260806 02:45:21.037065 3530 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:45:21.039537 3550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:21.041040 3530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:21.041200 3530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-0/wal
uuid: "ee07153e366241aa941ec4729842b4f4"
format_stamp: "Formatted at 2026-08-06 02:45:21 on dist-test-slave-gmjp"
server_key: "d98c29d6c3dc30365faf871c5ae5cb34"
server_key_iv: "5899c734865a7e171f95f403e5818f20"
server_key_version: "encryptionkey@0"
I20260806 02:45:21.041321 3530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:21.065840 3530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:21.069413 3530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:21.069649 3530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:21.071424 3530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:21.071497 3530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:21.071569 3530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:21.071607 3530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:45:21 dist-test-slave-gmjp krb5kdc[3359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984321, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:21.083280 3530 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.129:38407
I20260806 02:45:21.083365 3716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.129:38407 every 8 connection(s)
I20260806 02:45:21.084499 3530 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
I20260806 02:45:21.087411 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:21.072647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:60841 (local address 127.1.183.190:33023)
0806 02:45:21.072825 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:21.072829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:21.074029 (+ 1200us) server_negotiation.cc:407] Connection header received
0806 02:45:21.075423 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:21.075430 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:21.075508 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:21.075625 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:21.077373 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.078293 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.079689 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.080111 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.080253 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:21.083375 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:21.083379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:21.083382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:21.083412 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:21.085127 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:21.085748 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:21.085752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:21.085754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:21.085842 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:21.086187 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:21.086191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:21.086193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:21.086395 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:21.086504 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:21.087021 (+ 517us) server_negotiation.cc:299] Negotiation successful
0806 02:45:21.087220 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 02:45:21.088250 3563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:21.073245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:33023 (local address 127.1.183.129:60841)
0806 02:45:21.073848 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:21.073898 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:21.075072 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:21.075850 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:21.075865 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:21.076400 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:21.077168 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:21.077187 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.078445 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.078449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:21.079526 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:21.079538 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.080302 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.080306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:21.080457 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:21.081189 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:21.081215 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:21.083166 (+ 1951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:21.085253 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:21.085259 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:21.085275 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:21.085633 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:21.085946 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:21.085949 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:21.085951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:21.086091 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:21.086503 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:21.086510 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:21.086757 (+ 247us) client_negotiation.cc:769] Sending connection context
0806 02:45:21.086956 (+ 199us) client_negotiation.cc:240] Negotiation successful
0806 02:45:21.087348 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":132,"threads_started":1}
I20260806 02:45:21.088502 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 3530
I20260806 02:45:21.088624 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance
I20260806 02:45:21.088938 1758 external_mini_cluster.cc:1428] Setting key f3a603fce9f61a1c7585ad3670cfe11e
I20260806 02:45:21.089594 3718 heartbeater.cc:347] Connected to a master server at 127.1.183.190:33023
I20260806 02:45:21.089848 3718 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:21.092888 3718 heartbeater.cc:511] Master 127.1.183.190:33023 requested a full tablet report, sending...
I20260806 02:45:21.094672 3430 ts_manager.cc:194] Registered new tserver with Master: ee07153e366241aa941ec4729842b4f4 (127.1.183.129:38407)
I20260806 02:45:21.096122 3430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.129@KRBTEST.COM'} at 127.1.183.129:60841
WARNING: no policy specified for kudu/127.1.183.130@KRBTEST.COM; defaulting to no policy
I20260806 02:45:21.101151 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:21.092544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:39259 (local address 127.1.183.190:33023)
0806 02:45:21.092676 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:21.092681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:21.092961 (+ 280us) server_negotiation.cc:407] Connection header received
0806 02:45:21.093175 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:21.093180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:21.093241 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:21.093333 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:21.093925 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.094663 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.095827 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.096325 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.096457 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:21.097822 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:21.097826 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:21.097829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:21.097865 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:21.099452 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:21.099902 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:21.099906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:21.099908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:21.099988 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:21.100316 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:21.100319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:21.100321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:21.100496 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:21.100607 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:21.100866 (+ 259us) server_negotiation.cc:299] Negotiation successful
0806 02:45:21.100985 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":52}
I20260806 02:45:21.101161 3728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:21.092447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:33023 (local address 127.1.183.129:39259)
0806 02:45:21.092834 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:21.092861 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:21.093069 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:21.093366 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:21.093371 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:21.093596 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:21.093770 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:21.093778 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.094853 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.094856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:21.095670 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:21.095681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.096447 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.096451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:21.096542 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:21.097169 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:21.097184 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:21.097663 (+ 479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:21.099566 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:21.099571 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:21.099574 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:21.099796 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:21.100082 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:21.100086 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:21.100088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:21.100220 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:21.100596 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:21.100599 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:21.100696 (+ 97us) client_negotiation.cc:769] Sending connection context
0806 02:45:21.100802 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 02:45:21.100952 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":143,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
I20260806 02:45:21.121412 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.130:0
--local_ip_for_outbound_sockets=127.1.183.130
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:33023
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:41929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:21.243191 3731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:21.243461 3731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:21.243508 3731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:21.247648 3731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:21.247720 3731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:21.247761 3731 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:45:21.247794 3731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.130
I20260806 02:45:21.253286 3731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:41929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.183.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.183.190:33023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.130
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:21.254388 3731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:21.255401 3731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:21.266650 3740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:21.266651 3737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:21.266803 3736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:21.269925 3731 server_base.cc:1034] running on GCE node
I20260806 02:45:21.270527 3731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:21.271884 3731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:21.273217 3731 hybrid_clock.cc:648] HybridClock initialized: now 1785984321273143 us; error 118 us; skew 500 ppm
Aug 06 02:45:21 dist-test-slave-gmjp krb5kdc[3359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984321, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:21.277057 3731 init.cc:377] Logged in from keytab as kudu/127.1.183.130@KRBTEST.COM (short username kudu)
I20260806 02:45:21.278357 3731 webserver.cc:460] Webserver started at http://127.1.183.130:34439/ using document root <none> and password file <none>
I20260806 02:45:21.279016 3731 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:21.279119 3731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:21.279372 3731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:21.281769 3731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-1/data/instance:
uuid: "c80ac931d30b4479abf23fef8257d5dc"
format_stamp: "Formatted at 2026-08-06 02:45:21 on dist-test-slave-gmjp"
server_key: "e9fe0485850a89abe7cab6dd846f55d9"
server_key_iv: "9274eaffbc1ea029ee1fbc1770f7494b"
server_key_version: "encryptionkey@0"
I20260806 02:45:21.282410 3731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance:
uuid: "c80ac931d30b4479abf23fef8257d5dc"
format_stamp: "Formatted at 2026-08-06 02:45:21 on dist-test-slave-gmjp"
server_key: "e9fe0485850a89abe7cab6dd846f55d9"
server_key_iv: "9274eaffbc1ea029ee1fbc1770f7494b"
server_key_version: "encryptionkey@0"
I20260806 02:45:21.286710 3731 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:45:21.289574 3757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:21.291029 3731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:45:21.291163 3731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-1/wal
uuid: "c80ac931d30b4479abf23fef8257d5dc"
format_stamp: "Formatted at 2026-08-06 02:45:21 on dist-test-slave-gmjp"
server_key: "e9fe0485850a89abe7cab6dd846f55d9"
server_key_iv: "9274eaffbc1ea029ee1fbc1770f7494b"
server_key_version: "encryptionkey@0"
I20260806 02:45:21.291287 3731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:21.314170 3731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:21.317378 3731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:21.317597 3731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:21.319173 3731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:21.319223 3731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:21.319267 3731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:21.319321 3731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:45:21 dist-test-slave-gmjp krb5kdc[3359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984321, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:21.330791 3731 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.130:39975
I20260806 02:45:21.330896 3938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.130:39975 every 8 connection(s)
I20260806 02:45:21.331759 3731 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
I20260806 02:45:21.335490 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:21.320259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:48027 (local address 127.1.183.190:33023)
0806 02:45:21.320405 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:21.320409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:21.321591 (+ 1182us) server_negotiation.cc:407] Connection header received
0806 02:45:21.322915 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:21.322918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:21.322983 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:21.323095 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:21.324965 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.325879 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.327849 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.328090 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.328272 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:21.331565 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:21.331569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:21.331572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:21.331605 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:21.333263 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:21.333840 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:21.333844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:21.333845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:21.333948 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:21.334309 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:21.334313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:21.334315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:21.334531 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:21.334641 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:21.335141 (+ 500us) server_negotiation.cc:299] Negotiation successful
0806 02:45:21.335276 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:45:21.336452 3772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:21.320851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:33023 (local address 127.1.183.130:48027)
0806 02:45:21.321359 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:21.321487 (+ 128us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:21.322639 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:21.323327 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:21.323337 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:21.323940 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:21.324773 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:21.324796 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.326034 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.326040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:21.327684 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:21.327694 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.328317 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.328321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:21.328576 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:21.329391 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:21.329427 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:21.331365 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:21.333426 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:21.333434 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:21.333448 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:21.333714 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:21.334050 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:21.334053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:21.334055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:21.334195 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:21.334640 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:21.334647 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:21.334896 (+ 249us) client_negotiation.cc:769] Sending connection context
0806 02:45:21.335096 (+ 200us) client_negotiation.cc:240] Negotiation successful
0806 02:45:21.335438 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":145,"threads_started":1}
I20260806 02:45:21.337677 3940 heartbeater.cc:347] Connected to a master server at 127.1.183.190:33023
I20260806 02:45:21.337841 3940 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:21.338471 3940 heartbeater.cc:511] Master 127.1.183.190:33023 requested a full tablet report, sending...
I20260806 02:45:21.339658 3429 ts_manager.cc:194] Registered new tserver with Master: c80ac931d30b4479abf23fef8257d5dc (127.1.183.130:39975)
I20260806 02:45:21.340489 3429 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.130@KRBTEST.COM'} at 127.1.183.130:48027
I20260806 02:45:21.341331 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 3731
I20260806 02:45:21.341457 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance
I20260806 02:45:21.341735 1758 external_mini_cluster.cc:1428] Setting key c3d42eafaf20a381cde09cf7ae457ff3
I20260806 02:45:21.347733 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:21.340265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:46031 (local address 127.1.183.190:33023)
0806 02:45:21.340392 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:21.340397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:21.340558 (+ 161us) server_negotiation.cc:407] Connection header received
0806 02:45:21.340700 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:21.340704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:21.340771 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:21.340856 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:21.341393 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.341890 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.342860 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.343087 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.343192 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:21.344367 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:21.344375 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:21.344378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:21.344399 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:21.346039 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:21.346481 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:21.346484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:21.346485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:21.346559 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:21.346838 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:21.346842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:21.346844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:21.347068 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:21.347176 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:21.347450 (+ 274us) server_negotiation.cc:299] Negotiation successful
0806 02:45:21.347569 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 02:45:21.347745 3949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:21.340187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:33023 (local address 127.1.183.130:46031)
0806 02:45:21.340476 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:21.340489 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:21.340601 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:21.340897 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:21.340900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:21.341160 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:21.341276 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:21.341282 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.342015 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.342021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:21.342732 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:21.342742 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.343197 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.343201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:21.343291 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:21.343839 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:21.343850 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:21.344239 (+ 389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:21.346154 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:21.346157 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:21.346159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:21.346369 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:21.346657 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:21.346660 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:21.346662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:21.346749 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:21.347168 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:21.347171 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:21.347275 (+ 104us) client_negotiation.cc:769] Sending connection context
0806 02:45:21.347386 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 02:45:21.347553 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for kudu/127.1.183.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
I20260806 02:45:21.373766 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.131:0
--local_ip_for_outbound_sockets=127.1.183.131
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:33023
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:41929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:21.496028 3953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:21.496356 3953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:21.496409 3953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:21.500550 3953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:21.500612 3953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:21.500653 3953 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:45:21.500685 3953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.131
I20260806 02:45:21.506084 3953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:41929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.183.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.183.190:33023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.131
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:21.507129 3953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:21.508121 3953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:21.518209 3961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:21.518384 3953 server_base.cc:1034] running on GCE node
W20260806 02:45:21.518404 3966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:21.518209 3962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:21.518939 3953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:21.519655 3953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:21.520838 3953 hybrid_clock.cc:648] HybridClock initialized: now 1785984321520801 us; error 48 us; skew 500 ppm
Aug 06 02:45:21 dist-test-slave-gmjp krb5kdc[3359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984321, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:21.523790 3953 init.cc:377] Logged in from keytab as kudu/127.1.183.131@KRBTEST.COM (short username kudu)
I20260806 02:45:21.524751 3953 webserver.cc:460] Webserver started at http://127.1.183.131:34473/ using document root <none> and password file <none>
I20260806 02:45:21.525326 3953 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:21.525429 3953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:21.525655 3953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:21.527717 3953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-2/data/instance:
uuid: "098d000e12be4f308c3e7418a1c68f4d"
format_stamp: "Formatted at 2026-08-06 02:45:21 on dist-test-slave-gmjp"
server_key: "01c602969e4a8e6493caf193ba8e6632"
server_key_iv: "a9461d766ebbe66ef04a8f7b33bd30b8"
server_key_version: "encryptionkey@0"
I20260806 02:45:21.528332 3953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance:
uuid: "098d000e12be4f308c3e7418a1c68f4d"
format_stamp: "Formatted at 2026-08-06 02:45:21 on dist-test-slave-gmjp"
server_key: "01c602969e4a8e6493caf193ba8e6632"
server_key_iv: "a9461d766ebbe66ef04a8f7b33bd30b8"
server_key_version: "encryptionkey@0"
I20260806 02:45:21.532091 3953 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:21.534592 3983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:21.535809 3953 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 02:45:21.535956 3953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-2/wal
uuid: "098d000e12be4f308c3e7418a1c68f4d"
format_stamp: "Formatted at 2026-08-06 02:45:21 on dist-test-slave-gmjp"
server_key: "01c602969e4a8e6493caf193ba8e6632"
server_key_iv: "a9461d766ebbe66ef04a8f7b33bd30b8"
server_key_version: "encryptionkey@0"
I20260806 02:45:21.536068 3953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:21.548939 3953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:21.552493 3953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:21.552700 3953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:21.554200 3953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:21.554271 3953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:21.554347 3953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:21.554385 3953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:45:21 dist-test-slave-gmjp krb5kdc[3359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984321, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:21.565757 3953 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.131:46355
I20260806 02:45:21.565826 4148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.131:46355 every 8 connection(s)
I20260806 02:45:21.566855 3953 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
I20260806 02:45:21.569536 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:21.555150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:34387 (local address 127.1.183.190:33023)
0806 02:45:21.555279 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:21.555292 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:21.556431 (+ 1139us) server_negotiation.cc:407] Connection header received
0806 02:45:21.557516 (+ 1085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:21.557523 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:21.557595 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:21.557704 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:21.559294 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.560274 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.562005 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.562411 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.562522 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:21.565680 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:21.565684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:21.565687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:21.565721 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:21.567387 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:21.567989 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:21.567992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:21.567993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:21.568077 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:21.568449 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:21.568451 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:21.568452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:21.568641 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:21.568722 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:21.569230 (+ 508us) server_negotiation.cc:299] Negotiation successful
0806 02:45:21.569342 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 02:45:21.570568 3993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:21.555711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:33023 (local address 127.1.183.131:34387)
0806 02:45:21.556274 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:21.556328 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:21.557270 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:21.557928 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:21.557939 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:21.558429 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:21.559104 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:21.559127 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.560416 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.560420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:21.561878 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:21.561889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.562580 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.562583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:21.562719 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:21.563360 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:21.563386 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:21.565431 (+ 2045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:21.567539 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:21.567545 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:21.567561 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:21.567904 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:21.568146 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:21.568191 (+ 45us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:21.568196 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:21.568353 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:21.568735 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:21.568740 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:21.569009 (+ 269us) client_negotiation.cc:769] Sending connection context
0806 02:45:21.569168 (+ 159us) client_negotiation.cc:240] Negotiation successful
0806 02:45:21.569474 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":148,"threads_started":1}
I20260806 02:45:21.571615 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 3953
I20260806 02:45:21.571753 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance
I20260806 02:45:21.571887 4151 heartbeater.cc:347] Connected to a master server at 127.1.183.190:33023
I20260806 02:45:21.572079 1758 external_mini_cluster.cc:1428] Setting key 2bec28bcb460a44eb9e0dbb990a44c18
I20260806 02:45:21.572180 4151 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:21.572768 4151 heartbeater.cc:511] Master 127.1.183.190:33023 requested a full tablet report, sending...
I20260806 02:45:21.573899 3429 ts_manager.cc:194] Registered new tserver with Master: 098d000e12be4f308c3e7418a1c68f4d (127.1.183.131:46355)
I20260806 02:45:21.574649 3429 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.131@KRBTEST.COM'} at 127.1.183.131:34387
I20260806 02:45:21.581710 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:21.574579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:33023 (local address 127.1.183.131:54293)
0806 02:45:21.574940 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:21.574956 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:21.575057 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:21.575284 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:21.575290 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:21.575491 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:21.575598 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:21.575604 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.576606 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.576608 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:21.577168 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:21.577176 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.577595 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.577598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:21.577668 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:21.578177 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:21.578188 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:21.578548 (+ 360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:21.580284 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:21.580287 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:21.580289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:21.580515 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:21.580790 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:21.580793 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:21.580795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:21.580901 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:21.581264 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:21.581267 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:21.581353 (+ 86us) client_negotiation.cc:769] Sending connection context
0806 02:45:21.581438 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 02:45:21.581560 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":122,"threads_started":1}
I20260806 02:45:21.581735 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:21.574647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:54293 (local address 127.1.183.190:33023)
0806 02:45:21.574773 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:21.574780 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:21.575012 (+ 232us) server_negotiation.cc:407] Connection header received
0806 02:45:21.575142 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:21.575144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:21.575179 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:21.575242 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:21.575712 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.576467 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.577282 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.577495 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.577575 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:21.578670 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:21.578672 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:21.578674 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:21.578696 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:21.580122 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:21.580605 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:21.580607 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:21.580608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:21.580681 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:21.581002 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:21.581004 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:21.581005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:21.581180 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:21.581255 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:21.581514 (+ 259us) server_negotiation.cc:299] Negotiation successful
0806 02:45:21.581615 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 02:45:21.586287 1758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:45:21 dist-test-slave-gmjp krb5kdc[3359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984321, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:45:21 dist-test-slave-gmjp krb5kdc[3359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984321, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:21.613643 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:21.605331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38766 (local address 127.1.183.190:33023)
0806 02:45:21.605504 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:21.605508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:21.605523 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:45:21.605569 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:21.605572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:21.605613 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:21.605699 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:21.606384 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.606944 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.607747 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.607997 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.608088 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:21.610630 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:21.610633 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:21.610635 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:21.610661 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:21.612032 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:21.612455 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:21.612458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:21.612459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:21.612539 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:21.612796 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:21.612799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:21.612800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:21.613025 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:21.613154 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:21.613404 (+ 250us) server_negotiation.cc:299] Negotiation successful
0806 02:45:21.613513 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:45:21.616725 3429 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:45:21.619709 3429 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:45:21.634590 4186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:21.628627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:39975 (local address 127.0.0.1:37988)
0806 02:45:21.629539 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:21.629600 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:21.629828 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:21.630574 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:21.630583 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:21.630619 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:21.630862 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:21.630874 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.631870 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.631875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:21.633181 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:21.633190 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.634037 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.634041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:21.634146 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:21.634190 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 02:45:21.634302 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 02:45:21.634424 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"mutex_wait_us":60,"spinlock_wait_cycles":6912,"thread_start_us":77,"threads_started":1}
I20260806 02:45:21.634603 4185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:21.628787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37988 (local address 127.1.183.130:39975)
0806 02:45:21.629272 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:21.629286 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:21.629699 (+ 413us) server_negotiation.cc:407] Connection header received
0806 02:45:21.630025 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:21.630033 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:21.630262 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:21.630393 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:21.630993 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.631751 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.633312 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.633912 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.634024 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:21.634205 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:21.634347 (+ 142us) server_negotiation.cc:299] Negotiation successful
0806 02:45:21.634450 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":128,"threads_started":1}
I20260806 02:45:21.635378 4189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:21.629466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46394 (local address 127.1.183.131:46355)
0806 02:45:21.630010 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:21.630023 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:21.630365 (+ 342us) server_negotiation.cc:407] Connection header received
0806 02:45:21.630441 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:21.630446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:21.630622 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:21.630769 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:21.631035 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.632299 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.633970 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.634765 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.634873 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:21.635021 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:21.635139 (+ 118us) server_negotiation.cc:299] Negotiation successful
0806 02:45:21.635238 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":117,"threads_started":1}
I20260806 02:45:21.635588 4183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:21.628719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:38407 (local address 127.0.0.1:58200)
0806 02:45:21.630032 (+ 1313us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:21.630051 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:21.630160 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:21.630573 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:21.630579 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:21.630616 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:21.630892 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:21.630899 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.632354 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.632360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:21.633819 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:21.633831 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.635205 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.635208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:21.635292 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:21.635312 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 02:45:21.635382 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 02:45:21.635445 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1210,"mutex_wait_us":223,"spinlock_wait_cycles":5632,"thread_start_us":102,"threads_started":1}
I20260806 02:45:21.635929 4184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:21.629003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:46355 (local address 127.0.0.1:46394)
0806 02:45:21.630149 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:21.630161 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:21.630262 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:21.630744 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:21.630747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:21.630761 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:21.630892 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:21.630899 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.632425 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.632428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:21.633820 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:21.633831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.634899 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.634902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:21.635014 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:21.635036 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:45:21.635110 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 02:45:21.635192 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1084,"mutex_wait_us":136,"thread_start_us":100,"threads_started":1}
I20260806 02:45:21.636042 4188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:21.629448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58200 (local address 127.1.183.129:38407)
0806 02:45:21.629851 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:21.629860 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:21.630139 (+ 279us) server_negotiation.cc:407] Connection header received
0806 02:45:21.630271 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:21.630277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:21.630425 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:21.630567 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:21.631036 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.632218 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.633968 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.634893 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.635003 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:21.635145 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:21.635781 (+ 636us) server_negotiation.cc:299] Negotiation successful
0806 02:45:21.635887 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":100,"threads_started":1}
I20260806 02:45:21.638286 3831 tablet_service.cc:1467] Processing CreateTablet for tablet ea4c227ea4984032b59572a8a1874cfb (DEFAULT_TABLE table=test-table [id=46846a93e19648509eac85523569f107]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:21.639469 3831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea4c227ea4984032b59572a8a1874cfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:21.639004 3625 tablet_service.cc:1467] Processing CreateTablet for tablet ea4c227ea4984032b59572a8a1874cfb (DEFAULT_TABLE table=test-table [id=46846a93e19648509eac85523569f107]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:21.639015 4053 tablet_service.cc:1467] Processing CreateTablet for tablet ea4c227ea4984032b59572a8a1874cfb (DEFAULT_TABLE table=test-table [id=46846a93e19648509eac85523569f107]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:21.640100 3625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea4c227ea4984032b59572a8a1874cfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:21.640213 4053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea4c227ea4984032b59572a8a1874cfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:21.647109 4190 tablet_bootstrap.cc:492] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc: Bootstrap starting.
I20260806 02:45:21.648459 4191 tablet_bootstrap.cc:492] T ea4c227ea4984032b59572a8a1874cfb P ee07153e366241aa941ec4729842b4f4: Bootstrap starting.
I20260806 02:45:21.649712 4190 tablet_bootstrap.cc:654] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:21.650470 4191 tablet_bootstrap.cc:654] T ea4c227ea4984032b59572a8a1874cfb P ee07153e366241aa941ec4729842b4f4: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:21.650642 4190 log.cc:824] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:21.651177 4191 log.cc:824] T ea4c227ea4984032b59572a8a1874cfb P ee07153e366241aa941ec4729842b4f4: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:21.651368 4192 tablet_bootstrap.cc:492] T ea4c227ea4984032b59572a8a1874cfb P 098d000e12be4f308c3e7418a1c68f4d: Bootstrap starting.
I20260806 02:45:21.652805 4190 tablet_bootstrap.cc:492] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc: No bootstrap required, opened a new log
I20260806 02:45:21.653014 4190 ts_tablet_manager.cc:1397] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:45:21.653189 4191 tablet_bootstrap.cc:492] T ea4c227ea4984032b59572a8a1874cfb P ee07153e366241aa941ec4729842b4f4: No bootstrap required, opened a new log
I20260806 02:45:21.653298 4192 tablet_bootstrap.cc:654] T ea4c227ea4984032b59572a8a1874cfb P 098d000e12be4f308c3e7418a1c68f4d: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:21.653373 4191 ts_tablet_manager.cc:1397] T ea4c227ea4984032b59572a8a1874cfb P ee07153e366241aa941ec4729842b4f4: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260806 02:45:21.653954 4192 log.cc:824] T ea4c227ea4984032b59572a8a1874cfb P 098d000e12be4f308c3e7418a1c68f4d: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:21.655684 4192 tablet_bootstrap.cc:492] T ea4c227ea4984032b59572a8a1874cfb P 098d000e12be4f308c3e7418a1c68f4d: No bootstrap required, opened a new log
I20260806 02:45:21.655849 4192 ts_tablet_manager.cc:1397] T ea4c227ea4984032b59572a8a1874cfb P 098d000e12be4f308c3e7418a1c68f4d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 02:45:21.657286 4190 raft_consensus.cc:348] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } }
I20260806 02:45:21.657572 4190 raft_consensus.cc:374] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:21.657634 4190 raft_consensus.cc:729] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c80ac931d30b4479abf23fef8257d5dc, State: Initialized, Role: FOLLOWER
I20260806 02:45:21.657933 4191 raft_consensus.cc:348] T ea4c227ea4984032b59572a8a1874cfb P ee07153e366241aa941ec4729842b4f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } }
I20260806 02:45:21.658079 4190 consensus_queue.cc:260] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } }
I20260806 02:45:21.658181 4191 raft_consensus.cc:374] T ea4c227ea4984032b59572a8a1874cfb P ee07153e366241aa941ec4729842b4f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:21.658234 4191 raft_consensus.cc:729] T ea4c227ea4984032b59572a8a1874cfb P ee07153e366241aa941ec4729842b4f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee07153e366241aa941ec4729842b4f4, State: Initialized, Role: FOLLOWER
I20260806 02:45:21.658663 4191 consensus_queue.cc:260] T ea4c227ea4984032b59572a8a1874cfb P ee07153e366241aa941ec4729842b4f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } }
I20260806 02:45:21.658885 3940 heartbeater.cc:503] Master 127.1.183.190:33023 was elected leader, sending a full tablet report...
I20260806 02:45:21.659158 4190 ts_tablet_manager.cc:1428] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:45:21.659148 4192 raft_consensus.cc:348] T ea4c227ea4984032b59572a8a1874cfb P 098d000e12be4f308c3e7418a1c68f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } }
I20260806 02:45:21.659368 4192 raft_consensus.cc:374] T ea4c227ea4984032b59572a8a1874cfb P 098d000e12be4f308c3e7418a1c68f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:21.659430 4192 raft_consensus.cc:729] T ea4c227ea4984032b59572a8a1874cfb P 098d000e12be4f308c3e7418a1c68f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 098d000e12be4f308c3e7418a1c68f4d, State: Initialized, Role: FOLLOWER
I20260806 02:45:21.659529 4191 ts_tablet_manager.cc:1428] T ea4c227ea4984032b59572a8a1874cfb P ee07153e366241aa941ec4729842b4f4: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 02:45:21.659796 4192 consensus_queue.cc:260] T ea4c227ea4984032b59572a8a1874cfb P 098d000e12be4f308c3e7418a1c68f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } }
I20260806 02:45:21.659973 3718 heartbeater.cc:503] Master 127.1.183.190:33023 was elected leader, sending a full tablet report...
I20260806 02:45:21.660509 4151 heartbeater.cc:503] Master 127.1.183.190:33023 was elected leader, sending a full tablet report...
I20260806 02:45:21.660769 4192 ts_tablet_manager.cc:1428] T ea4c227ea4984032b59572a8a1874cfb P 098d000e12be4f308c3e7418a1c68f4d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 02:45:21.803881 4201 raft_consensus.cc:482] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:45:21.804121 4201 raft_consensus.cc:504] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } }
I20260806 02:45:21.805428 4201 leader_election.cc:288] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee07153e366241aa941ec4729842b4f4 (127.1.183.129:38407), 098d000e12be4f308c3e7418a1c68f4d (127.1.183.131:46355)
I20260806 02:45:21.809356 4188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:21.805827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:45959 (local address 127.1.183.129:38407)
0806 02:45:21.805971 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:21.805976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:21.806078 (+ 102us) server_negotiation.cc:407] Connection header received
0806 02:45:21.806196 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:21.806199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:21.806249 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:21.806330 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:21.806626 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.807149 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.808249 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.808681 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.808817 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:21.808878 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:21.809146 (+ 268us) server_negotiation.cc:299] Negotiation successful
0806 02:45:21.809227 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:45:21.809356 3772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:21.805679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:38407 (local address 127.1.183.130:45959)
0806 02:45:21.805946 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:21.805964 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:21.806088 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:21.806343 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:21.806345 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:21.806362 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:21.806507 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:21.806512 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.807259 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.807265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:21.808075 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:21.808084 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.808819 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.808823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:21.809009 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:21.809028 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 02:45:21.809126 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 02:45:21.809210 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":30}
I20260806 02:45:21.810174 4204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:21.805747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:46355 (local address 127.1.183.130:36295)
0806 02:45:21.806170 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:21.806181 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:21.806318 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:21.806586 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:21.806589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:21.806605 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:21.806739 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:21.806744 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.807480 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.807485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:21.808987 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:21.808998 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.809799 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.809803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:21.809897 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:21.809932 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 02:45:21.810003 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 02:45:21.810057 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":27,"thread_start_us":139,"threads_started":1}
I20260806 02:45:21.810248 4189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:21.805974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:36295 (local address 127.1.183.131:46355)
0806 02:45:21.806097 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:21.806101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:21.806279 (+ 178us) server_negotiation.cc:407] Connection header received
0806 02:45:21.806432 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:21.806435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:21.806483 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:21.806562 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:21.806854 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.807365 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.809122 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.809653 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.809790 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:21.809833 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:21.810045 (+ 212us) server_negotiation.cc:299] Negotiation successful
0806 02:45:21.810128 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:45:21.810284 3653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea4c227ea4984032b59572a8a1874cfb" candidate_uuid: "c80ac931d30b4479abf23fef8257d5dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee07153e366241aa941ec4729842b4f4" is_pre_election: true
I20260806 02:45:21.810665 3653 raft_consensus.cc:2454] T ea4c227ea4984032b59572a8a1874cfb P ee07153e366241aa941ec4729842b4f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c80ac931d30b4479abf23fef8257d5dc in term 0.
I20260806 02:45:21.810868 4084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea4c227ea4984032b59572a8a1874cfb" candidate_uuid: "c80ac931d30b4479abf23fef8257d5dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098d000e12be4f308c3e7418a1c68f4d" is_pre_election: true
I20260806 02:45:21.811151 4084 raft_consensus.cc:2454] T ea4c227ea4984032b59572a8a1874cfb P 098d000e12be4f308c3e7418a1c68f4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c80ac931d30b4479abf23fef8257d5dc in term 0.
I20260806 02:45:21.811271 3763 leader_election.cc:302] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c80ac931d30b4479abf23fef8257d5dc, ee07153e366241aa941ec4729842b4f4; no voters:
I20260806 02:45:21.811607 4201 raft_consensus.cc:2781] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:45:21.811700 4201 raft_consensus.cc:482] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:45:21.811762 4201 raft_consensus.cc:3037] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:21.812878 4201 raft_consensus.cc:504] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } }
I20260806 02:45:21.813247 4201 leader_election.cc:288] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc [CANDIDATE]: Term 1 election: Requested vote from peers ee07153e366241aa941ec4729842b4f4 (127.1.183.129:38407), 098d000e12be4f308c3e7418a1c68f4d (127.1.183.131:46355)
I20260806 02:45:21.813678 4084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea4c227ea4984032b59572a8a1874cfb" candidate_uuid: "c80ac931d30b4479abf23fef8257d5dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098d000e12be4f308c3e7418a1c68f4d"
I20260806 02:45:21.813683 3653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea4c227ea4984032b59572a8a1874cfb" candidate_uuid: "c80ac931d30b4479abf23fef8257d5dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee07153e366241aa941ec4729842b4f4"
I20260806 02:45:21.813817 4084 raft_consensus.cc:3037] T ea4c227ea4984032b59572a8a1874cfb P 098d000e12be4f308c3e7418a1c68f4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:21.813796 3653 raft_consensus.cc:3037] T ea4c227ea4984032b59572a8a1874cfb P ee07153e366241aa941ec4729842b4f4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:21.814961 3653 raft_consensus.cc:2454] T ea4c227ea4984032b59572a8a1874cfb P ee07153e366241aa941ec4729842b4f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c80ac931d30b4479abf23fef8257d5dc in term 1.
I20260806 02:45:21.815189 4084 raft_consensus.cc:2454] T ea4c227ea4984032b59572a8a1874cfb P 098d000e12be4f308c3e7418a1c68f4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c80ac931d30b4479abf23fef8257d5dc in term 1.
I20260806 02:45:21.815356 3763 leader_election.cc:302] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c80ac931d30b4479abf23fef8257d5dc, ee07153e366241aa941ec4729842b4f4; no voters:
I20260806 02:45:21.815551 4201 raft_consensus.cc:2781] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:21.815804 4201 raft_consensus.cc:686] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc [term 1 LEADER]: Becoming Leader. State: Replica: c80ac931d30b4479abf23fef8257d5dc, State: Running, Role: LEADER
I20260806 02:45:21.816190 4201 consensus_queue.cc:237] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } }
W20260806 02:45:21.817873 4153 tablet.cc:2369] T ea4c227ea4984032b59572a8a1874cfb P 098d000e12be4f308c3e7418a1c68f4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:45:21.819758 3429 catalog_manager.cc:5495] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc reported cstate change: term changed from 0 to 1, leader changed from <none> to c80ac931d30b4479abf23fef8257d5dc (127.1.183.130). New cstate: current_term: 1 leader_uuid: "c80ac931d30b4479abf23fef8257d5dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } health_report { overall_health: UNKNOWN } } }
W20260806 02:45:21.832938 3942 tablet.cc:2369] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:45:21.835475 4185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:21.831767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38002 (local address 127.1.183.130:39975)
0806 02:45:21.831895 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:21.831898 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:21.831913 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:45:21.832103 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:21.832106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:21.832232 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:21.832321 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:21.832574 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.833304 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.834151 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:21.834371 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:21.834452 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:21.834729 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:21.835158 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:21.835245 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:21.835312 (+ 67us) server_negotiation.cc:299] Negotiation successful
0806 02:45:21.835365 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260806 02:45:21.835939 3719 tablet.cc:2369] T ea4c227ea4984032b59572a8a1874cfb P ee07153e366241aa941ec4729842b4f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:45:21.841709 3653 raft_consensus.cc:1261] T ea4c227ea4984032b59572a8a1874cfb P ee07153e366241aa941ec4729842b4f4 [term 1 FOLLOWER]: Refusing update from remote peer c80ac931d30b4479abf23fef8257d5dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:21.841709 4084 raft_consensus.cc:1261] T ea4c227ea4984032b59572a8a1874cfb P 098d000e12be4f308c3e7418a1c68f4d [term 1 FOLLOWER]: Refusing update from remote peer c80ac931d30b4479abf23fef8257d5dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:21.842442 4201 consensus_queue.cc:1035] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:21.842648 4205 consensus_queue.cc:1035] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:21.849692 4210 mvcc.cc:205] Tried to move back new op lower bound from 7315391782255972352 to 7315391782162354176. Current Snapshot: MvccSnapshot[applied={T|T < 7315391782255972352}]
I20260806 02:45:21.852914 4213 mvcc.cc:205] Tried to move back new op lower bound from 7315391782255972352 to 7315391782162354176. Current Snapshot: MvccSnapshot[applied={T|T < 7315391782255972352}]
I20260806 02:45:21.853595 4211 mvcc.cc:205] Tried to move back new op lower bound from 7315391782255972352 to 7315391782162354176. Current Snapshot: MvccSnapshot[applied={T|T < 7315391782255972352 or (T in {7315391782255972352})}]
I20260806 02:45:21.864818 3429 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:21.865036 3429 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:21.867867 3429 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1154321aa1f74423a374e00d8cef960c: Sending DeleteTablet for 3 replicas of tablet ea4c227ea4984032b59572a8a1874cfb
I20260806 02:45:21.868840 3831 tablet_service.cc:1514] Processing DeleteTablet for tablet ea4c227ea4984032b59572a8a1874cfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:21 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:37988
I20260806 02:45:21.868953 3625 tablet_service.cc:1514] Processing DeleteTablet for tablet ea4c227ea4984032b59572a8a1874cfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:21 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:58200
I20260806 02:45:21.869536 4223 tablet_replica.cc:330] T ea4c227ea4984032b59572a8a1874cfb P ee07153e366241aa941ec4729842b4f4: stopping tablet replica
I20260806 02:45:21.869691 4053 tablet_service.cc:1514] Processing DeleteTablet for tablet ea4c227ea4984032b59572a8a1874cfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:21 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:46394
I20260806 02:45:21.869910 4222 tablet_replica.cc:330] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc: stopping tablet replica
I20260806 02:45:21.869920 4223 raft_consensus.cc:2229] T ea4c227ea4984032b59572a8a1874cfb P ee07153e366241aa941ec4729842b4f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:21.870309 4222 raft_consensus.cc:2229] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:45:21.870237 3429 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 02:45:21.870380 4223 raft_consensus.cc:2258] T ea4c227ea4984032b59572a8a1874cfb P ee07153e366241aa941ec4729842b4f4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260806 02:45:21.870752 3429 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:45:21.870937 4222 raft_consensus.cc:2258] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:21.872328 4225 tablet_replica.cc:330] T ea4c227ea4984032b59572a8a1874cfb P 098d000e12be4f308c3e7418a1c68f4d: stopping tablet replica
I20260806 02:45:21.872480 4222 ts_tablet_manager.cc:1905] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:21.872649 4225 raft_consensus.cc:2229] T ea4c227ea4984032b59572a8a1874cfb P 098d000e12be4f308c3e7418a1c68f4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:21.872979 4225 raft_consensus.cc:2258] T ea4c227ea4984032b59572a8a1874cfb P 098d000e12be4f308c3e7418a1c68f4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:21.875509 4222 ts_tablet_manager.cc:1918] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:45:21.875633 4222 log.cc:1196] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-1/wal/wals/ea4c227ea4984032b59572a8a1874cfb
I20260806 02:45:21.875958 4222 ts_tablet_manager.cc:1939] T ea4c227ea4984032b59572a8a1874cfb P c80ac931d30b4479abf23fef8257d5dc: Deleting consensus metadata
I20260806 02:45:21.876209 3625 tablet_service.cc:1467] Processing CreateTablet for tablet e75d3eb5013b4b83a1bc22e6ac2ff699 (DEFAULT_TABLE table=test-table [id=ad5becae1a5c41c182f6267075e46807]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:21.876515 3625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e75d3eb5013b4b83a1bc22e6ac2ff699. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:21.876674 3831 tablet_service.cc:1467] Processing CreateTablet for tablet e75d3eb5013b4b83a1bc22e6ac2ff699 (DEFAULT_TABLE table=test-table [id=ad5becae1a5c41c182f6267075e46807]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:21.876932 3831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e75d3eb5013b4b83a1bc22e6ac2ff699. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:21.878533 4225 ts_tablet_manager.cc:1905] T ea4c227ea4984032b59572a8a1874cfb P 098d000e12be4f308c3e7418a1c68f4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:21.879309 4053 tablet_service.cc:1467] Processing CreateTablet for tablet e75d3eb5013b4b83a1bc22e6ac2ff699 (DEFAULT_TABLE table=test-table [id=ad5becae1a5c41c182f6267075e46807]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:21.879586 4053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e75d3eb5013b4b83a1bc22e6ac2ff699. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:21.879828 4191 tablet_bootstrap.cc:492] T e75d3eb5013b4b83a1bc22e6ac2ff699 P ee07153e366241aa941ec4729842b4f4: Bootstrap starting.
I20260806 02:45:21.880877 4191 tablet_bootstrap.cc:654] T e75d3eb5013b4b83a1bc22e6ac2ff699 P ee07153e366241aa941ec4729842b4f4: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:21.881558 4225 ts_tablet_manager.cc:1918] T ea4c227ea4984032b59572a8a1874cfb P 098d000e12be4f308c3e7418a1c68f4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:45:21.881716 4225 log.cc:1196] T ea4c227ea4984032b59572a8a1874cfb P 098d000e12be4f308c3e7418a1c68f4d: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-2/wal/wals/ea4c227ea4984032b59572a8a1874cfb
I20260806 02:45:21.882097 4192 tablet_bootstrap.cc:492] T e75d3eb5013b4b83a1bc22e6ac2ff699 P 098d000e12be4f308c3e7418a1c68f4d: Bootstrap starting.
I20260806 02:45:21.882180 4190 tablet_bootstrap.cc:492] T e75d3eb5013b4b83a1bc22e6ac2ff699 P c80ac931d30b4479abf23fef8257d5dc: Bootstrap starting.
I20260806 02:45:21.882200 4225 ts_tablet_manager.cc:1939] T ea4c227ea4984032b59572a8a1874cfb P 098d000e12be4f308c3e7418a1c68f4d: Deleting consensus metadata
I20260806 02:45:21.883086 4192 tablet_bootstrap.cc:654] T e75d3eb5013b4b83a1bc22e6ac2ff699 P 098d000e12be4f308c3e7418a1c68f4d: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:21.883102 4190 tablet_bootstrap.cc:654] T e75d3eb5013b4b83a1bc22e6ac2ff699 P c80ac931d30b4479abf23fef8257d5dc: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:21.883983 3409 catalog_manager.cc:4841] TS 098d000e12be4f308c3e7418a1c68f4d (127.1.183.131:46355): tablet ea4c227ea4984032b59572a8a1874cfb (table test-table [id=46846a93e19648509eac85523569f107]) successfully deleted
I20260806 02:45:21.884282 4192 tablet_bootstrap.cc:492] T e75d3eb5013b4b83a1bc22e6ac2ff699 P 098d000e12be4f308c3e7418a1c68f4d: No bootstrap required, opened a new log
I20260806 02:45:21.884373 4192 ts_tablet_manager.cc:1397] T e75d3eb5013b4b83a1bc22e6ac2ff699 P 098d000e12be4f308c3e7418a1c68f4d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:45:21.884378 3405 catalog_manager.cc:4841] TS c80ac931d30b4479abf23fef8257d5dc (127.1.183.130:39975): tablet ea4c227ea4984032b59572a8a1874cfb (table test-table [id=46846a93e19648509eac85523569f107]) successfully deleted
I20260806 02:45:21.884666 4190 tablet_bootstrap.cc:492] T e75d3eb5013b4b83a1bc22e6ac2ff699 P c80ac931d30b4479abf23fef8257d5dc: No bootstrap required, opened a new log
I20260806 02:45:21.884748 4190 ts_tablet_manager.cc:1397] T e75d3eb5013b4b83a1bc22e6ac2ff699 P c80ac931d30b4479abf23fef8257d5dc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 02:45:21.884811 4192 raft_consensus.cc:348] T e75d3eb5013b4b83a1bc22e6ac2ff699 P 098d000e12be4f308c3e7418a1c68f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } }
I20260806 02:45:21.884917 4192 raft_consensus.cc:374] T e75d3eb5013b4b83a1bc22e6ac2ff699 P 098d000e12be4f308c3e7418a1c68f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:21.884948 4192 raft_consensus.cc:729] T e75d3eb5013b4b83a1bc22e6ac2ff699 P 098d000e12be4f308c3e7418a1c68f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 098d000e12be4f308c3e7418a1c68f4d, State: Initialized, Role: FOLLOWER
I20260806 02:45:21.885068 4192 consensus_queue.cc:260] T e75d3eb5013b4b83a1bc22e6ac2ff699 P 098d000e12be4f308c3e7418a1c68f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } }
I20260806 02:45:21.885304 4192 ts_tablet_manager.cc:1428] T e75d3eb5013b4b83a1bc22e6ac2ff699 P 098d000e12be4f308c3e7418a1c68f4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:45:21.885234 4190 raft_consensus.cc:348] T e75d3eb5013b4b83a1bc22e6ac2ff699 P c80ac931d30b4479abf23fef8257d5dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } }
I20260806 02:45:21.885367 4190 raft_consensus.cc:374] T e75d3eb5013b4b83a1bc22e6ac2ff699 P c80ac931d30b4479abf23fef8257d5dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:21.885411 4190 raft_consensus.cc:729] T e75d3eb5013b4b83a1bc22e6ac2ff699 P c80ac931d30b4479abf23fef8257d5dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c80ac931d30b4479abf23fef8257d5dc, State: Initialized, Role: FOLLOWER
I20260806 02:45:21.885434 4223 ts_tablet_manager.cc:1905] T ea4c227ea4984032b59572a8a1874cfb P ee07153e366241aa941ec4729842b4f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:21.885543 4190 consensus_queue.cc:260] T e75d3eb5013b4b83a1bc22e6ac2ff699 P c80ac931d30b4479abf23fef8257d5dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } }
I20260806 02:45:21.885787 4190 ts_tablet_manager.cc:1428] T e75d3eb5013b4b83a1bc22e6ac2ff699 P c80ac931d30b4479abf23fef8257d5dc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 02:45:21.888051 4191 tablet_bootstrap.cc:492] T e75d3eb5013b4b83a1bc22e6ac2ff699 P ee07153e366241aa941ec4729842b4f4: No bootstrap required, opened a new log
I20260806 02:45:21.888206 4191 ts_tablet_manager.cc:1397] T e75d3eb5013b4b83a1bc22e6ac2ff699 P ee07153e366241aa941ec4729842b4f4: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260806 02:45:21.888703 4191 raft_consensus.cc:348] T e75d3eb5013b4b83a1bc22e6ac2ff699 P ee07153e366241aa941ec4729842b4f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } }
I20260806 02:45:21.888820 4191 raft_consensus.cc:374] T e75d3eb5013b4b83a1bc22e6ac2ff699 P ee07153e366241aa941ec4729842b4f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:21.888851 4191 raft_consensus.cc:729] T e75d3eb5013b4b83a1bc22e6ac2ff699 P ee07153e366241aa941ec4729842b4f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee07153e366241aa941ec4729842b4f4, State: Initialized, Role: FOLLOWER
I20260806 02:45:21.888974 4191 consensus_queue.cc:260] T e75d3eb5013b4b83a1bc22e6ac2ff699 P ee07153e366241aa941ec4729842b4f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } }
I20260806 02:45:21.889185 4223 ts_tablet_manager.cc:1918] T ea4c227ea4984032b59572a8a1874cfb P ee07153e366241aa941ec4729842b4f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:45:21.889321 4223 log.cc:1196] T ea4c227ea4984032b59572a8a1874cfb P ee07153e366241aa941ec4729842b4f4: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-0/wal/wals/ea4c227ea4984032b59572a8a1874cfb
I20260806 02:45:21.889223 4191 ts_tablet_manager.cc:1428] T e75d3eb5013b4b83a1bc22e6ac2ff699 P ee07153e366241aa941ec4729842b4f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:45:21.889729 4223 ts_tablet_manager.cc:1939] T ea4c227ea4984032b59572a8a1874cfb P ee07153e366241aa941ec4729842b4f4: Deleting consensus metadata
I20260806 02:45:21.891013 3407 catalog_manager.cc:4841] TS ee07153e366241aa941ec4729842b4f4 (127.1.183.129:38407): tablet ea4c227ea4984032b59572a8a1874cfb (table test-table [id=46846a93e19648509eac85523569f107]) successfully deleted
I20260806 02:45:22.016687 4202 raft_consensus.cc:482] T e75d3eb5013b4b83a1bc22e6ac2ff699 P ee07153e366241aa941ec4729842b4f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:45:22.016921 4202 raft_consensus.cc:504] T e75d3eb5013b4b83a1bc22e6ac2ff699 P ee07153e366241aa941ec4729842b4f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } }
I20260806 02:45:22.018137 4202 leader_election.cc:288] T e75d3eb5013b4b83a1bc22e6ac2ff699 P ee07153e366241aa941ec4729842b4f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c80ac931d30b4479abf23fef8257d5dc (127.1.183.130:39975), 098d000e12be4f308c3e7418a1c68f4d (127.1.183.131:46355)
I20260806 02:45:22.022940 4230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:22.018393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:39975 (local address 127.1.183.129:50447)
0806 02:45:22.018813 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:22.018830 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:22.018950 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:22.019405 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:22.019407 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:22.019425 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:22.019572 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:22.019579 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.020581 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.020585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:22.021683 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:22.021700 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.022461 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.022465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:22.022580 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:22.022607 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 02:45:22.022689 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 02:45:22.022766 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":146,"threads_started":1}
I20260806 02:45:22.022957 4185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:22.018708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:50447 (local address 127.1.183.130:39975)
0806 02:45:22.018890 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:22.018895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:22.018914 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:45:22.019110 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:22.019115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:22.019181 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:22.019236 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:22.019728 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.020449 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.021815 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.022302 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.022438 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:22.022498 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:22.022726 (+ 228us) server_negotiation.cc:299] Negotiation successful
0806 02:45:22.022823 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:45:22.023680 3869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e75d3eb5013b4b83a1bc22e6ac2ff699" candidate_uuid: "ee07153e366241aa941ec4729842b4f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c80ac931d30b4479abf23fef8257d5dc" is_pre_election: true
I20260806 02:45:22.023747 4189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:22.018892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:35883 (local address 127.1.183.131:46355)
0806 02:45:22.019046 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:22.019051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:22.019236 (+ 185us) server_negotiation.cc:407] Connection header received
0806 02:45:22.019338 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:22.019341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:22.019407 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:22.019494 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:22.020021 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.020660 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.022151 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.022722 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.022849 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:22.022896 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:22.023554 (+ 658us) server_negotiation.cc:299] Negotiation successful
0806 02:45:22.023630 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 02:45:22.023777 4231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:22.018475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:46355 (local address 127.1.183.129:35883)
0806 02:45:22.019132 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:22.019144 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:22.019248 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:22.019519 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:22.019524 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:22.019543 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:22.019645 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:22.019898 (+ 253us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.020808 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.020812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:22.022021 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:22.022034 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.022963 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.022969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:22.023423 (+ 454us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:22.023452 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 02:45:22.023533 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 02:45:22.023600 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":92,"thread_start_us":119,"threads_started":1}
I20260806 02:45:22.023968 3869 raft_consensus.cc:2454] T e75d3eb5013b4b83a1bc22e6ac2ff699 P c80ac931d30b4479abf23fef8257d5dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee07153e366241aa941ec4729842b4f4 in term 0.
I20260806 02:45:22.024250 4084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e75d3eb5013b4b83a1bc22e6ac2ff699" candidate_uuid: "ee07153e366241aa941ec4729842b4f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098d000e12be4f308c3e7418a1c68f4d" is_pre_election: true
I20260806 02:45:22.024431 4084 raft_consensus.cc:2454] T e75d3eb5013b4b83a1bc22e6ac2ff699 P 098d000e12be4f308c3e7418a1c68f4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee07153e366241aa941ec4729842b4f4 in term 0.
I20260806 02:45:22.024577 3551 leader_election.cc:302] T e75d3eb5013b4b83a1bc22e6ac2ff699 P ee07153e366241aa941ec4729842b4f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c80ac931d30b4479abf23fef8257d5dc, ee07153e366241aa941ec4729842b4f4; no voters:
I20260806 02:45:22.024854 4202 raft_consensus.cc:2781] T e75d3eb5013b4b83a1bc22e6ac2ff699 P ee07153e366241aa941ec4729842b4f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:45:22.024911 4202 raft_consensus.cc:482] T e75d3eb5013b4b83a1bc22e6ac2ff699 P ee07153e366241aa941ec4729842b4f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:45:22.024940 4202 raft_consensus.cc:3037] T e75d3eb5013b4b83a1bc22e6ac2ff699 P ee07153e366241aa941ec4729842b4f4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:22.026013 4202 raft_consensus.cc:504] T e75d3eb5013b4b83a1bc22e6ac2ff699 P ee07153e366241aa941ec4729842b4f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } }
I20260806 02:45:22.026384 4202 leader_election.cc:288] T e75d3eb5013b4b83a1bc22e6ac2ff699 P ee07153e366241aa941ec4729842b4f4 [CANDIDATE]: Term 1 election: Requested vote from peers c80ac931d30b4479abf23fef8257d5dc (127.1.183.130:39975), 098d000e12be4f308c3e7418a1c68f4d (127.1.183.131:46355)
I20260806 02:45:22.026752 4084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e75d3eb5013b4b83a1bc22e6ac2ff699" candidate_uuid: "ee07153e366241aa941ec4729842b4f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098d000e12be4f308c3e7418a1c68f4d"
I20260806 02:45:22.026893 4084 raft_consensus.cc:3037] T e75d3eb5013b4b83a1bc22e6ac2ff699 P 098d000e12be4f308c3e7418a1c68f4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:22.026940 3869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e75d3eb5013b4b83a1bc22e6ac2ff699" candidate_uuid: "ee07153e366241aa941ec4729842b4f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c80ac931d30b4479abf23fef8257d5dc"
I20260806 02:45:22.027105 3869 raft_consensus.cc:3037] T e75d3eb5013b4b83a1bc22e6ac2ff699 P c80ac931d30b4479abf23fef8257d5dc [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:22.027892 4084 raft_consensus.cc:2454] T e75d3eb5013b4b83a1bc22e6ac2ff699 P 098d000e12be4f308c3e7418a1c68f4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee07153e366241aa941ec4729842b4f4 in term 1.
I20260806 02:45:22.028103 3869 raft_consensus.cc:2454] T e75d3eb5013b4b83a1bc22e6ac2ff699 P c80ac931d30b4479abf23fef8257d5dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee07153e366241aa941ec4729842b4f4 in term 1.
I20260806 02:45:22.028225 3555 leader_election.cc:302] T e75d3eb5013b4b83a1bc22e6ac2ff699 P ee07153e366241aa941ec4729842b4f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 098d000e12be4f308c3e7418a1c68f4d, ee07153e366241aa941ec4729842b4f4; no voters:
I20260806 02:45:22.028512 4202 raft_consensus.cc:2781] T e75d3eb5013b4b83a1bc22e6ac2ff699 P ee07153e366241aa941ec4729842b4f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:22.028759 4202 raft_consensus.cc:686] T e75d3eb5013b4b83a1bc22e6ac2ff699 P ee07153e366241aa941ec4729842b4f4 [term 1 LEADER]: Becoming Leader. State: Replica: ee07153e366241aa941ec4729842b4f4, State: Running, Role: LEADER
I20260806 02:45:22.029026 4202 consensus_queue.cc:237] T e75d3eb5013b4b83a1bc22e6ac2ff699 P ee07153e366241aa941ec4729842b4f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } }
I20260806 02:45:22.031464 3429 catalog_manager.cc:5495] T e75d3eb5013b4b83a1bc22e6ac2ff699 P ee07153e366241aa941ec4729842b4f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee07153e366241aa941ec4729842b4f4 (127.1.183.129). New cstate: current_term: 1 leader_uuid: "ee07153e366241aa941ec4729842b4f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } health_report { overall_health: UNKNOWN } } }
Aug 06 02:45:22 dist-test-slave-gmjp krb5kdc[3359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984320, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:22.114539 4243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:22.106102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:33023 (local address 127.0.0.1:38772)
0806 02:45:22.106408 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:22.106421 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:22.106565 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:22.106816 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:22.106818 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:22.107206 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:22.107325 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:22.107331 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.108045 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.108048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:22.108633 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:22.108642 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.109067 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.109070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:22.109133 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:22.109707 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:22.109719 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:22.111189 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:22.112983 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:22.112992 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:22.112998 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:22.113258 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:22.113543 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:22.113546 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:22.113547 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:22.113635 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:22.113996 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:22.114001 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:22.114109 (+ 108us) client_negotiation.cc:769] Sending connection context
0806 02:45:22.114202 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 02:45:22.114351 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":115,"threads_started":1}
I20260806 02:45:22.114540 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:22.106197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38772 (local address 127.1.183.190:33023)
0806 02:45:22.106334 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:22.106339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:22.106499 (+ 160us) server_negotiation.cc:407] Connection header received
0806 02:45:22.106652 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:22.106655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:22.106696 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:22.106766 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:22.107431 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.107941 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.108749 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.108969 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.109046 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:22.111322 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:22.111325 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:22.111327 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:22.111348 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:22.112860 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:22.113365 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:22.113368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:22.113369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:22.113441 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:22.113717 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:22.113719 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:22.113720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:22.113889 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:22.113978 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:22.114257 (+ 279us) server_negotiation.cc:299] Negotiation successful
0806 02:45:22.114410 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":55}
I20260806 02:45:22.119146 3429 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:38772:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 02:45:22.119998 3429 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:45:22.124926 3625 tablet_service.cc:1467] Processing CreateTablet for tablet 90e6d0f4b8f3497992b933f4c263eb45 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c17db675f7f04691aeaa384c222e2d39]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:45:22.125005 3831 tablet_service.cc:1467] Processing CreateTablet for tablet 90e6d0f4b8f3497992b933f4c263eb45 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c17db675f7f04691aeaa384c222e2d39]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:45:22.125168 4053 tablet_service.cc:1467] Processing CreateTablet for tablet 90e6d0f4b8f3497992b933f4c263eb45 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c17db675f7f04691aeaa384c222e2d39]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:45:22.125438 3625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90e6d0f4b8f3497992b933f4c263eb45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:22.125438 3831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90e6d0f4b8f3497992b933f4c263eb45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:22.125551 4053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90e6d0f4b8f3497992b933f4c263eb45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:22.127995 4191 tablet_bootstrap.cc:492] T 90e6d0f4b8f3497992b933f4c263eb45 P ee07153e366241aa941ec4729842b4f4: Bootstrap starting.
I20260806 02:45:22.128348 4190 tablet_bootstrap.cc:492] T 90e6d0f4b8f3497992b933f4c263eb45 P c80ac931d30b4479abf23fef8257d5dc: Bootstrap starting.
I20260806 02:45:22.128417 4192 tablet_bootstrap.cc:492] T 90e6d0f4b8f3497992b933f4c263eb45 P 098d000e12be4f308c3e7418a1c68f4d: Bootstrap starting.
I20260806 02:45:22.129065 4191 tablet_bootstrap.cc:654] T 90e6d0f4b8f3497992b933f4c263eb45 P ee07153e366241aa941ec4729842b4f4: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:22.129091 4190 tablet_bootstrap.cc:654] T 90e6d0f4b8f3497992b933f4c263eb45 P c80ac931d30b4479abf23fef8257d5dc: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:22.129431 4192 tablet_bootstrap.cc:654] T 90e6d0f4b8f3497992b933f4c263eb45 P 098d000e12be4f308c3e7418a1c68f4d: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:22.130429 4190 tablet_bootstrap.cc:492] T 90e6d0f4b8f3497992b933f4c263eb45 P c80ac931d30b4479abf23fef8257d5dc: No bootstrap required, opened a new log
I20260806 02:45:22.130534 4190 ts_tablet_manager.cc:1397] T 90e6d0f4b8f3497992b933f4c263eb45 P c80ac931d30b4479abf23fef8257d5dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 02:45:22.130709 4192 tablet_bootstrap.cc:492] T 90e6d0f4b8f3497992b933f4c263eb45 P 098d000e12be4f308c3e7418a1c68f4d: No bootstrap required, opened a new log
I20260806 02:45:22.130709 4191 tablet_bootstrap.cc:492] T 90e6d0f4b8f3497992b933f4c263eb45 P ee07153e366241aa941ec4729842b4f4: No bootstrap required, opened a new log
I20260806 02:45:22.130800 4191 ts_tablet_manager.cc:1397] T 90e6d0f4b8f3497992b933f4c263eb45 P ee07153e366241aa941ec4729842b4f4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 02:45:22.130800 4192 ts_tablet_manager.cc:1397] T 90e6d0f4b8f3497992b933f4c263eb45 P 098d000e12be4f308c3e7418a1c68f4d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:45:22.130977 4190 raft_consensus.cc:348] T 90e6d0f4b8f3497992b933f4c263eb45 P c80ac931d30b4479abf23fef8257d5dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } }
I20260806 02:45:22.131098 4190 raft_consensus.cc:374] T 90e6d0f4b8f3497992b933f4c263eb45 P c80ac931d30b4479abf23fef8257d5dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:22.131137 4190 raft_consensus.cc:729] T 90e6d0f4b8f3497992b933f4c263eb45 P c80ac931d30b4479abf23fef8257d5dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c80ac931d30b4479abf23fef8257d5dc, State: Initialized, Role: FOLLOWER
I20260806 02:45:22.131238 4191 raft_consensus.cc:348] T 90e6d0f4b8f3497992b933f4c263eb45 P ee07153e366241aa941ec4729842b4f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } }
I20260806 02:45:22.131238 4192 raft_consensus.cc:348] T 90e6d0f4b8f3497992b933f4c263eb45 P 098d000e12be4f308c3e7418a1c68f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } }
I20260806 02:45:22.131280 4190 consensus_queue.cc:260] T 90e6d0f4b8f3497992b933f4c263eb45 P c80ac931d30b4479abf23fef8257d5dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } }
I20260806 02:45:22.131358 4191 raft_consensus.cc:374] T 90e6d0f4b8f3497992b933f4c263eb45 P ee07153e366241aa941ec4729842b4f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:22.131358 4192 raft_consensus.cc:374] T 90e6d0f4b8f3497992b933f4c263eb45 P 098d000e12be4f308c3e7418a1c68f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:22.131403 4192 raft_consensus.cc:729] T 90e6d0f4b8f3497992b933f4c263eb45 P 098d000e12be4f308c3e7418a1c68f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 098d000e12be4f308c3e7418a1c68f4d, State: Initialized, Role: FOLLOWER
I20260806 02:45:22.131403 4191 raft_consensus.cc:729] T 90e6d0f4b8f3497992b933f4c263eb45 P ee07153e366241aa941ec4729842b4f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee07153e366241aa941ec4729842b4f4, State: Initialized, Role: FOLLOWER
I20260806 02:45:22.131533 4191 consensus_queue.cc:260] T 90e6d0f4b8f3497992b933f4c263eb45 P ee07153e366241aa941ec4729842b4f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } }
I20260806 02:45:22.131533 4192 consensus_queue.cc:260] T 90e6d0f4b8f3497992b933f4c263eb45 P 098d000e12be4f308c3e7418a1c68f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } }
I20260806 02:45:22.131589 4190 ts_tablet_manager.cc:1428] T 90e6d0f4b8f3497992b933f4c263eb45 P c80ac931d30b4479abf23fef8257d5dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:45:22.131799 4191 ts_tablet_manager.cc:1428] T 90e6d0f4b8f3497992b933f4c263eb45 P ee07153e366241aa941ec4729842b4f4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 02:45:22.131807 4192 ts_tablet_manager.cc:1428] T 90e6d0f4b8f3497992b933f4c263eb45 P 098d000e12be4f308c3e7418a1c68f4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:45:22.131867 4201 tablet_replica.cc:439] T 90e6d0f4b8f3497992b933f4c263eb45 P c80ac931d30b4479abf23fef8257d5dc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } } }
I20260806 02:45:22.132011 4201 tablet_replica.cc:442] T 90e6d0f4b8f3497992b933f4c263eb45 P c80ac931d30b4479abf23fef8257d5dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:22.131999 4203 tablet_replica.cc:439] T 90e6d0f4b8f3497992b933f4c263eb45 P 098d000e12be4f308c3e7418a1c68f4d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } } }
I20260806 02:45:22.132140 4203 tablet_replica.cc:442] T 90e6d0f4b8f3497992b933f4c263eb45 P 098d000e12be4f308c3e7418a1c68f4d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:22.132221 4202 tablet_replica.cc:439] T 90e6d0f4b8f3497992b933f4c263eb45 P ee07153e366241aa941ec4729842b4f4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } } }
I20260806 02:45:22.132367 4202 tablet_replica.cc:442] T 90e6d0f4b8f3497992b933f4c263eb45 P ee07153e366241aa941ec4729842b4f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:22.311420 4201 raft_consensus.cc:482] T 90e6d0f4b8f3497992b933f4c263eb45 P c80ac931d30b4479abf23fef8257d5dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:45:22.311610 4201 raft_consensus.cc:504] T 90e6d0f4b8f3497992b933f4c263eb45 P c80ac931d30b4479abf23fef8257d5dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } }
I20260806 02:45:22.312095 4201 leader_election.cc:288] T 90e6d0f4b8f3497992b933f4c263eb45 P c80ac931d30b4479abf23fef8257d5dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee07153e366241aa941ec4729842b4f4 (127.1.183.129:38407), 098d000e12be4f308c3e7418a1c68f4d (127.1.183.131:46355)
I20260806 02:45:22.312683 4084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e6d0f4b8f3497992b933f4c263eb45" candidate_uuid: "c80ac931d30b4479abf23fef8257d5dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098d000e12be4f308c3e7418a1c68f4d" is_pre_election: true
I20260806 02:45:22.312685 3653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e6d0f4b8f3497992b933f4c263eb45" candidate_uuid: "c80ac931d30b4479abf23fef8257d5dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee07153e366241aa941ec4729842b4f4" is_pre_election: true
I20260806 02:45:22.312857 4084 raft_consensus.cc:2454] T 90e6d0f4b8f3497992b933f4c263eb45 P 098d000e12be4f308c3e7418a1c68f4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c80ac931d30b4479abf23fef8257d5dc in term 0.
I20260806 02:45:22.312870 3653 raft_consensus.cc:2454] T 90e6d0f4b8f3497992b933f4c263eb45 P ee07153e366241aa941ec4729842b4f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c80ac931d30b4479abf23fef8257d5dc in term 0.
I20260806 02:45:22.313222 3764 leader_election.cc:302] T 90e6d0f4b8f3497992b933f4c263eb45 P c80ac931d30b4479abf23fef8257d5dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 098d000e12be4f308c3e7418a1c68f4d, c80ac931d30b4479abf23fef8257d5dc; no voters:
I20260806 02:45:22.313423 4201 raft_consensus.cc:2781] T 90e6d0f4b8f3497992b933f4c263eb45 P c80ac931d30b4479abf23fef8257d5dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:45:22.313490 4201 raft_consensus.cc:482] T 90e6d0f4b8f3497992b933f4c263eb45 P c80ac931d30b4479abf23fef8257d5dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:45:22.313516 4201 raft_consensus.cc:3037] T 90e6d0f4b8f3497992b933f4c263eb45 P c80ac931d30b4479abf23fef8257d5dc [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:22.314574 4201 raft_consensus.cc:504] T 90e6d0f4b8f3497992b933f4c263eb45 P c80ac931d30b4479abf23fef8257d5dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } }
I20260806 02:45:22.314950 4201 leader_election.cc:288] T 90e6d0f4b8f3497992b933f4c263eb45 P c80ac931d30b4479abf23fef8257d5dc [CANDIDATE]: Term 1 election: Requested vote from peers ee07153e366241aa941ec4729842b4f4 (127.1.183.129:38407), 098d000e12be4f308c3e7418a1c68f4d (127.1.183.131:46355)
I20260806 02:45:22.315310 3653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e6d0f4b8f3497992b933f4c263eb45" candidate_uuid: "c80ac931d30b4479abf23fef8257d5dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee07153e366241aa941ec4729842b4f4"
I20260806 02:45:22.315459 3653 raft_consensus.cc:3037] T 90e6d0f4b8f3497992b933f4c263eb45 P ee07153e366241aa941ec4729842b4f4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:22.315470 4084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e6d0f4b8f3497992b933f4c263eb45" candidate_uuid: "c80ac931d30b4479abf23fef8257d5dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098d000e12be4f308c3e7418a1c68f4d"
I20260806 02:45:22.315582 4084 raft_consensus.cc:3037] T 90e6d0f4b8f3497992b933f4c263eb45 P 098d000e12be4f308c3e7418a1c68f4d [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:22.316694 3653 raft_consensus.cc:2454] T 90e6d0f4b8f3497992b933f4c263eb45 P ee07153e366241aa941ec4729842b4f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c80ac931d30b4479abf23fef8257d5dc in term 1.
I20260806 02:45:22.316694 4084 raft_consensus.cc:2454] T 90e6d0f4b8f3497992b933f4c263eb45 P 098d000e12be4f308c3e7418a1c68f4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c80ac931d30b4479abf23fef8257d5dc in term 1.
I20260806 02:45:22.316999 3764 leader_election.cc:302] T 90e6d0f4b8f3497992b933f4c263eb45 P c80ac931d30b4479abf23fef8257d5dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 098d000e12be4f308c3e7418a1c68f4d, c80ac931d30b4479abf23fef8257d5dc; no voters:
I20260806 02:45:22.317186 4201 raft_consensus.cc:2781] T 90e6d0f4b8f3497992b933f4c263eb45 P c80ac931d30b4479abf23fef8257d5dc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:22.317323 4201 raft_consensus.cc:686] T 90e6d0f4b8f3497992b933f4c263eb45 P c80ac931d30b4479abf23fef8257d5dc [term 1 LEADER]: Becoming Leader. State: Replica: c80ac931d30b4479abf23fef8257d5dc, State: Running, Role: LEADER
I20260806 02:45:22.317476 4201 consensus_queue.cc:237] T 90e6d0f4b8f3497992b933f4c263eb45 P c80ac931d30b4479abf23fef8257d5dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } }
I20260806 02:45:22.318145 4205 tablet_replica.cc:439] T 90e6d0f4b8f3497992b933f4c263eb45 P c80ac931d30b4479abf23fef8257d5dc: TxnStatusTablet state changed. Reason: New leader c80ac931d30b4479abf23fef8257d5dc. Latest consensus state: current_term: 1 leader_uuid: "c80ac931d30b4479abf23fef8257d5dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } } }
I20260806 02:45:22.318257 4205 tablet_replica.cc:442] T 90e6d0f4b8f3497992b933f4c263eb45 P c80ac931d30b4479abf23fef8257d5dc: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:45:22.318580 4252 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 02:45:22.318828 4252 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 02:45:22.319347 3430 catalog_manager.cc:5495] T 90e6d0f4b8f3497992b933f4c263eb45 P c80ac931d30b4479abf23fef8257d5dc reported cstate change: term changed from 0 to 1, leader changed from <none> to c80ac931d30b4479abf23fef8257d5dc (127.1.183.130). New cstate: current_term: 1 leader_uuid: "c80ac931d30b4479abf23fef8257d5dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } health_report { overall_health: UNKNOWN } } }
I20260806 02:45:22.333099 4185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:22.329759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44666 (local address 127.1.183.130:39975)
0806 02:45:22.329881 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:22.329888 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:22.329904 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:45:22.330109 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:22.330112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:22.330162 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:22.330245 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:22.330499 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.331062 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.331995 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.332268 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.332367 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:22.332638 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:22.332716 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:22.332792 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:22.332917 (+ 125us) server_negotiation.cc:299] Negotiation successful
0806 02:45:22.332968 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:45:22.333102 4243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:22.329660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:39975 (local address 127.0.0.1:44666)
0806 02:45:22.329844 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:22.329861 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:22.329984 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:22.330279 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:22.330282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:22.330290 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:45:22.330396 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:22.330401 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.331171 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.331174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:22.331864 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:22.331873 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.332379 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.332383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:22.332463 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:22.332507 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:45:22.332797 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:45:22.332814 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 02:45:22.332889 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 02:45:22.332982 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 02:45:22.335945 4084 raft_consensus.cc:1261] T 90e6d0f4b8f3497992b933f4c263eb45 P 098d000e12be4f308c3e7418a1c68f4d [term 1 FOLLOWER]: Refusing update from remote peer c80ac931d30b4479abf23fef8257d5dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:22.335955 3653 raft_consensus.cc:1261] T 90e6d0f4b8f3497992b933f4c263eb45 P ee07153e366241aa941ec4729842b4f4 [term 1 FOLLOWER]: Refusing update from remote peer c80ac931d30b4479abf23fef8257d5dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:22.336403 4201 consensus_queue.cc:1035] T 90e6d0f4b8f3497992b933f4c263eb45 P c80ac931d30b4479abf23fef8257d5dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:22.336745 4205 consensus_queue.cc:1035] T 90e6d0f4b8f3497992b933f4c263eb45 P c80ac931d30b4479abf23fef8257d5dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:22.338089 4202 tablet_replica.cc:439] T 90e6d0f4b8f3497992b933f4c263eb45 P ee07153e366241aa941ec4729842b4f4: TxnStatusTablet state changed. Reason: New leader c80ac931d30b4479abf23fef8257d5dc. Latest consensus state: current_term: 1 leader_uuid: "c80ac931d30b4479abf23fef8257d5dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } } }
I20260806 02:45:22.338093 4203 tablet_replica.cc:439] T 90e6d0f4b8f3497992b933f4c263eb45 P 098d000e12be4f308c3e7418a1c68f4d: TxnStatusTablet state changed. Reason: New leader c80ac931d30b4479abf23fef8257d5dc. Latest consensus state: current_term: 1 leader_uuid: "c80ac931d30b4479abf23fef8257d5dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } } }
I20260806 02:45:22.338215 4202 tablet_replica.cc:442] T 90e6d0f4b8f3497992b933f4c263eb45 P ee07153e366241aa941ec4729842b4f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:22.338215 4203 tablet_replica.cc:442] T 90e6d0f4b8f3497992b933f4c263eb45 P 098d000e12be4f308c3e7418a1c68f4d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:22.346123 4205 tablet_replica.cc:439] T 90e6d0f4b8f3497992b933f4c263eb45 P c80ac931d30b4479abf23fef8257d5dc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c80ac931d30b4479abf23fef8257d5dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } } }
I20260806 02:45:22.346279 4205 tablet_replica.cc:442] T 90e6d0f4b8f3497992b933f4c263eb45 P c80ac931d30b4479abf23fef8257d5dc: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:45:22.346618 4205 tablet_replica.cc:439] T 90e6d0f4b8f3497992b933f4c263eb45 P c80ac931d30b4479abf23fef8257d5dc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c80ac931d30b4479abf23fef8257d5dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } } }
I20260806 02:45:22.346712 4205 tablet_replica.cc:442] T 90e6d0f4b8f3497992b933f4c263eb45 P c80ac931d30b4479abf23fef8257d5dc: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:45:22.347357 4202 tablet_replica.cc:439] T 90e6d0f4b8f3497992b933f4c263eb45 P ee07153e366241aa941ec4729842b4f4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c80ac931d30b4479abf23fef8257d5dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } } }
I20260806 02:45:22.347488 4202 tablet_replica.cc:442] T 90e6d0f4b8f3497992b933f4c263eb45 P ee07153e366241aa941ec4729842b4f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:22.349834 4203 tablet_replica.cc:439] T 90e6d0f4b8f3497992b933f4c263eb45 P 098d000e12be4f308c3e7418a1c68f4d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c80ac931d30b4479abf23fef8257d5dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee07153e366241aa941ec4729842b4f4" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 38407 } } peers { permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 } } peers { permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 } } }
I20260806 02:45:22.349979 4203 tablet_replica.cc:442] T 90e6d0f4b8f3497992b933f4c263eb45 P 098d000e12be4f308c3e7418a1c68f4d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:22.368147 4260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:22.353733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40226 (local address 127.1.183.129:38407)
0806 02:45:22.354276 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:22.354286 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:22.362879 (+ 8593us) server_negotiation.cc:407] Connection header received
0806 02:45:22.363020 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:22.363025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:22.363106 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:22.363260 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:22.363734 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.364530 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.365940 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.366376 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.366496 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:22.366842 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:22.367545 (+ 703us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:22.367665 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:22.367727 (+ 62us) server_negotiation.cc:299] Negotiation successful
0806 02:45:22.367790 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":130,"threads_started":1}
I20260806 02:45:22.384016 4185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:22.377164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:57507 (local address 127.1.183.130:39975)
0806 02:45:22.377299 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:22.377303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:22.380533 (+ 3230us) server_negotiation.cc:407] Connection header received
0806 02:45:22.380681 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:22.380684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:22.380736 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:22.380826 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:22.381118 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.381629 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.382879 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.383087 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.383161 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:22.383512 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:22.383601 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:22.383678 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:22.383805 (+ 127us) server_negotiation.cc:299] Negotiation successful
0806 02:45:22.383859 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 02:45:22.384048 4265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:22.377047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:39975 (local address 127.1.183.129:57507)
0806 02:45:22.380432 (+ 3385us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:22.380449 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:22.380583 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:22.380841 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:22.380844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:22.380858 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:45:22.381002 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:22.381008 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.381730 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.381734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:22.382763 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:22.382775 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.383197 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.383200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:22.383300 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:22.383361 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:45:22.383688 (+ 327us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:45:22.383698 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 02:45:22.383777 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 02:45:22.383863 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3247,"thread_start_us":108,"threads_started":1}
I20260806 02:45:22.391985 4084 raft_consensus.cc:1261] T e75d3eb5013b4b83a1bc22e6ac2ff699 P 098d000e12be4f308c3e7418a1c68f4d [term 1 FOLLOWER]: Refusing update from remote peer ee07153e366241aa941ec4729842b4f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:22.392275 3869 raft_consensus.cc:1261] T e75d3eb5013b4b83a1bc22e6ac2ff699 P c80ac931d30b4479abf23fef8257d5dc [term 1 FOLLOWER]: Refusing update from remote peer ee07153e366241aa941ec4729842b4f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:22.392539 4232 consensus_queue.cc:1035] T e75d3eb5013b4b83a1bc22e6ac2ff699 P ee07153e366241aa941ec4729842b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "098d000e12be4f308c3e7418a1c68f4d" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:22.392781 4202 consensus_queue.cc:1035] T e75d3eb5013b4b83a1bc22e6ac2ff699 P ee07153e366241aa941ec4729842b4f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c80ac931d30b4479abf23fef8257d5dc" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:22.419934 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:22.413516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:38407 (local address 127.1.183.130:43677)
0806 02:45:22.416400 (+ 2884us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:22.416426 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:22.416552 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:22.416813 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:22.416816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:22.416825 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:45:22.416935 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:22.416941 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.417690 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.417694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:22.418582 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:22.418595 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.419190 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.419197 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:22.419284 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:22.419353 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:45:22.419676 (+ 323us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:45:22.419682 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 02:45:22.419730 (+ 48us) client_negotiation.cc:240] Negotiation successful
0806 02:45:22.419803 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2765,"thread_start_us":127,"threads_started":1}
I20260806 02:45:22.419948 4260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:22.413600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:43677 (local address 127.1.183.129:38407)
0806 02:45:22.413721 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:22.413730 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:22.416503 (+ 2773us) server_negotiation.cc:407] Connection header received
0806 02:45:22.416666 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:22.416669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:22.416723 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:22.416801 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:22.417045 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.417577 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.418723 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.419090 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.419192 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:22.419505 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:22.419591 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:22.419666 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:22.419751 (+ 85us) server_negotiation.cc:299] Negotiation successful
0806 02:45:22.419810 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 02:45:22.454520 3430 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:22.454741 3430 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:22.457551 3430 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1154321aa1f74423a374e00d8cef960c: Sending DeleteTablet for 3 replicas of tablet e75d3eb5013b4b83a1bc22e6ac2ff699
I20260806 02:45:22.458356 3625 tablet_service.cc:1514] Processing DeleteTablet for tablet e75d3eb5013b4b83a1bc22e6ac2ff699 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:22 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:58200
I20260806 02:45:22.458578 3831 tablet_service.cc:1514] Processing DeleteTablet for tablet e75d3eb5013b4b83a1bc22e6ac2ff699 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:22 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:37988
I20260806 02:45:22.458935 4278 tablet_replica.cc:330] T e75d3eb5013b4b83a1bc22e6ac2ff699 P ee07153e366241aa941ec4729842b4f4: stopping tablet replica
I20260806 02:45:22.459129 4278 raft_consensus.cc:2229] T e75d3eb5013b4b83a1bc22e6ac2ff699 P ee07153e366241aa941ec4729842b4f4 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:45:22.459177 4053 tablet_service.cc:1514] Processing DeleteTablet for tablet e75d3eb5013b4b83a1bc22e6ac2ff699 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:22 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:46394
I20260806 02:45:22.459607 4278 raft_consensus.cc:2258] T e75d3eb5013b4b83a1bc22e6ac2ff699 P ee07153e366241aa941ec4729842b4f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:22.460486 4278 ts_tablet_manager.cc:1905] T e75d3eb5013b4b83a1bc22e6ac2ff699 P ee07153e366241aa941ec4729842b4f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:22.461030 4281 tablet_replica.cc:330] T e75d3eb5013b4b83a1bc22e6ac2ff699 P 098d000e12be4f308c3e7418a1c68f4d: stopping tablet replica
I20260806 02:45:22.461180 4281 raft_consensus.cc:2229] T e75d3eb5013b4b83a1bc22e6ac2ff699 P 098d000e12be4f308c3e7418a1c68f4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:22.461323 4281 raft_consensus.cc:2258] T e75d3eb5013b4b83a1bc22e6ac2ff699 P 098d000e12be4f308c3e7418a1c68f4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:22.462088 4281 ts_tablet_manager.cc:1905] T e75d3eb5013b4b83a1bc22e6ac2ff699 P 098d000e12be4f308c3e7418a1c68f4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:22.464340 4278 ts_tablet_manager.cc:1918] T e75d3eb5013b4b83a1bc22e6ac2ff699 P ee07153e366241aa941ec4729842b4f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:45:22.464628 4280 tablet_replica.cc:330] T e75d3eb5013b4b83a1bc22e6ac2ff699 P c80ac931d30b4479abf23fef8257d5dc: stopping tablet replica
I20260806 02:45:22.464749 4278 log.cc:1196] T e75d3eb5013b4b83a1bc22e6ac2ff699 P ee07153e366241aa941ec4729842b4f4: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-0/wal/wals/e75d3eb5013b4b83a1bc22e6ac2ff699
I20260806 02:45:22.464957 4280 raft_consensus.cc:2229] T e75d3eb5013b4b83a1bc22e6ac2ff699 P c80ac931d30b4479abf23fef8257d5dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:22.465337 4278 ts_tablet_manager.cc:1939] T e75d3eb5013b4b83a1bc22e6ac2ff699 P ee07153e366241aa941ec4729842b4f4: Deleting consensus metadata
I20260806 02:45:22.465538 4280 raft_consensus.cc:2258] T e75d3eb5013b4b83a1bc22e6ac2ff699 P c80ac931d30b4479abf23fef8257d5dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:22.465626 4281 ts_tablet_manager.cc:1918] T e75d3eb5013b4b83a1bc22e6ac2ff699 P 098d000e12be4f308c3e7418a1c68f4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:45:22.465754 4281 log.cc:1196] T e75d3eb5013b4b83a1bc22e6ac2ff699 P 098d000e12be4f308c3e7418a1c68f4d: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-2/wal/wals/e75d3eb5013b4b83a1bc22e6ac2ff699
I20260806 02:45:22.466053 4281 ts_tablet_manager.cc:1939] T e75d3eb5013b4b83a1bc22e6ac2ff699 P 098d000e12be4f308c3e7418a1c68f4d: Deleting consensus metadata
I20260806 02:45:22.466589 4280 ts_tablet_manager.cc:1905] T e75d3eb5013b4b83a1bc22e6ac2ff699 P c80ac931d30b4479abf23fef8257d5dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:22.466934 3409 catalog_manager.cc:4841] TS 098d000e12be4f308c3e7418a1c68f4d (127.1.183.131:46355): tablet e75d3eb5013b4b83a1bc22e6ac2ff699 (table test-table [id=ad5becae1a5c41c182f6267075e46807]) successfully deleted
I20260806 02:45:22.467512 3407 catalog_manager.cc:4841] TS ee07153e366241aa941ec4729842b4f4 (127.1.183.129:38407): tablet e75d3eb5013b4b83a1bc22e6ac2ff699 (table test-table [id=ad5becae1a5c41c182f6267075e46807]) successfully deleted
I20260806 02:45:22.470517 4280 ts_tablet_manager.cc:1918] T e75d3eb5013b4b83a1bc22e6ac2ff699 P c80ac931d30b4479abf23fef8257d5dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:45:22.470767 4280 log.cc:1196] T e75d3eb5013b4b83a1bc22e6ac2ff699 P c80ac931d30b4479abf23fef8257d5dc: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984313887056-1758-0/minicluster-data/ts-1/wal/wals/e75d3eb5013b4b83a1bc22e6ac2ff699
I20260806 02:45:22.471136 4280 ts_tablet_manager.cc:1939] T e75d3eb5013b4b83a1bc22e6ac2ff699 P c80ac931d30b4479abf23fef8257d5dc: Deleting consensus metadata
I20260806 02:45:22.472100 3405 catalog_manager.cc:4841] TS c80ac931d30b4479abf23fef8257d5dc (127.1.183.130:39975): tablet e75d3eb5013b4b83a1bc22e6ac2ff699 (table test-table [id=ad5becae1a5c41c182f6267075e46807]) successfully deleted
Aug 06 02:45:22 dist-test-slave-gmjp krb5kdc[3359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984321, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.183.129@KRBTEST.COM
I20260806 02:45:22.482275 4260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:22.468645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40230 (local address 127.1.183.129:38407)
0806 02:45:22.468791 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:22.468796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:22.469479 (+ 683us) server_negotiation.cc:407] Connection header received
0806 02:45:22.469698 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:22.469701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:22.469756 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:22.469827 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:22.470878 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.471699 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.473032 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.473442 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.473549 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:22.477871 (+ 4322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:22.477885 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:22.477906 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:22.477939 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:22.480250 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:22.480759 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:22.480767 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:22.480769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:22.480880 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:22.481169 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:22.481172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:22.481174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:22.481520 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:22.481726 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:22.481945 (+ 219us) server_negotiation.cc:299] Negotiation successful
0806 02:45:22.482083 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260806 02:45:22.482960 3708 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40230
I20260806 02:45:22.494488 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:22.485059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34024 (local address 127.1.183.190:33023)
0806 02:45:22.485200 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:22.485204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:22.485394 (+ 190us) server_negotiation.cc:407] Connection header received
0806 02:45:22.485520 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:22.485523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:22.485580 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:22.485670 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:22.486444 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.487260 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.488395 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:22.488753 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:22.488858 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:22.490620 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:22.490624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:22.490627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:22.490658 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:22.492540 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:22.493054 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:22.493058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:22.493060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:22.493152 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:22.493446 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:22.493450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:22.493452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:22.493693 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:22.493828 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:22.494098 (+ 270us) server_negotiation.cc:299] Negotiation successful
0806 02:45:22.494229 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260806 02:45:22.494913 3430 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34024
I20260806 02:45:22.496814 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 3530
I20260806 02:45:22.514875 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 3731
I20260806 02:45:22.525636 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 3953
I20260806 02:45:22.538944 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 3382
2026-08-06T02:45:22Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3373 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:45:22 dist-test-slave-gmjp krb5kdc[4294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:45:22 dist-test-slave-gmjp krb5kdc[4294](info): set up 2 sockets
Aug 06 02:45:22 dist-test-slave-gmjp krb5kdc[4294](info): commencing operation
krb5kdc: starting...
W20260806 02:45:23.830364 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.260s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:45:23 dist-test-slave-gmjp krb5kdc[4294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:45:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
I20260806 02:45:23.967835 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:41751
--webserver_interface=127.1.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:36719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:41751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:24.085585 4312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:24.085949 4312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:24.086036 4312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:24.090296 4312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:24.090406 4312 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 02:45:24.090432 4312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:24.090467 4312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:24.090503 4312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:24.090544 4312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:24.096299 4312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:36719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:41751
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:41751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:24.097833 4312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:24.099033 4312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:24.108276 4318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:24.108446 4312 server_base.cc:1034] running on GCE node
W20260806 02:45:24.108285 4324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:24.108286 4321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:24.109164 4312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:24.110404 4312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:24.111614 4312 hybrid_clock.cc:648] HybridClock initialized: now 1785984324111582 us; error 49 us; skew 500 ppm
Aug 06 02:45:24 dist-test-slave-gmjp krb5kdc[4294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984324, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:24.119179 4312 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:45:24.120362 4312 webserver.cc:460] Webserver started at http://127.1.183.190:45827/ using document root <none> and password file <none>
I20260806 02:45:24.121017 4312 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:24.121119 4312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:24.121377 4312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:24.123695 4312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "612964c3ba9140fea152adcd8e22048c"
format_stamp: "Formatted at 2026-08-06 02:45:24 on dist-test-slave-gmjp"
server_key: "5ff107eb47d7dd91e7a4183250656dec"
server_key_iv: "2cd26d7e4f9fe2711d98388b1b7bc6e9"
server_key_version: "encryptionkey@0"
I20260806 02:45:24.124389 4312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "612964c3ba9140fea152adcd8e22048c"
format_stamp: "Formatted at 2026-08-06 02:45:24 on dist-test-slave-gmjp"
server_key: "5ff107eb47d7dd91e7a4183250656dec"
server_key_iv: "2cd26d7e4f9fe2711d98388b1b7bc6e9"
server_key_version: "encryptionkey@0"
I20260806 02:45:24.129132 4312 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:45:24.131762 4336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:24.133229 4312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:24.133400 4312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "612964c3ba9140fea152adcd8e22048c"
format_stamp: "Formatted at 2026-08-06 02:45:24 on dist-test-slave-gmjp"
server_key: "5ff107eb47d7dd91e7a4183250656dec"
server_key_iv: "2cd26d7e4f9fe2711d98388b1b7bc6e9"
server_key_version: "encryptionkey@0"
I20260806 02:45:24.133533 4312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:24.151513 4312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:24.155220 4312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:24.155469 4312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:24.164431 4312 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:41751
I20260806 02:45:24.164461 4429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:41751 every 8 connection(s)
I20260806 02:45:24.165524 4312 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:24.166232 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 4312
I20260806 02:45:24.166343 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:45:24.166580 1758 external_mini_cluster.cc:1428] Setting key 75db2dc16dfdf7bbcd8e32187a4f47c6
I20260806 02:45:24.168964 4431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:24.175195 4431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 612964c3ba9140fea152adcd8e22048c: Bootstrap starting.
Aug 06 02:45:24 dist-test-slave-gmjp krb5kdc[4294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:24.177964 4431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 612964c3ba9140fea152adcd8e22048c: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:24.179160 4431 log.cc:824] T 00000000000000000000000000000000 P 612964c3ba9140fea152adcd8e22048c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:24.181340 4435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:24.168048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59572 (local address 127.1.183.190:41751)
0806 02:45:24.168530 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:24.168546 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:24.168582 (+ 36us) server_negotiation.cc:407] Connection header received
0806 02:45:24.169313 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:24.169332 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:24.169696 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:24.170054 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:24.170913 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.171948 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.172921 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.173407 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.173579 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.176318 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:24.176324 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:24.176339 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:24.176372 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:24.178224 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:24.178706 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:24.178717 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:24.178722 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:24.178840 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:24.179135 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:24.179139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:24.179141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:24.179658 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:24.179893 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:24.180233 (+ 340us) server_negotiation.cc:299] Negotiation successful
0806 02:45:24.180557 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":147,"threads_started":1}
I20260806 02:45:24.181694 4431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 612964c3ba9140fea152adcd8e22048c: No bootstrap required, opened a new log
I20260806 02:45:24.185257 4431 raft_consensus.cc:348] T 00000000000000000000000000000000 P 612964c3ba9140fea152adcd8e22048c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "612964c3ba9140fea152adcd8e22048c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 41751 } }
I20260806 02:45:24.185506 4431 raft_consensus.cc:374] T 00000000000000000000000000000000 P 612964c3ba9140fea152adcd8e22048c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:24.185544 4431 raft_consensus.cc:729] T 00000000000000000000000000000000 P 612964c3ba9140fea152adcd8e22048c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 612964c3ba9140fea152adcd8e22048c, State: Initialized, Role: FOLLOWER
I20260806 02:45:24.185932 4431 consensus_queue.cc:260] T 00000000000000000000000000000000 P 612964c3ba9140fea152adcd8e22048c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "612964c3ba9140fea152adcd8e22048c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 41751 } }
I20260806 02:45:24.186048 4431 raft_consensus.cc:388] T 00000000000000000000000000000000 P 612964c3ba9140fea152adcd8e22048c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:24.186098 4431 raft_consensus.cc:482] T 00000000000000000000000000000000 P 612964c3ba9140fea152adcd8e22048c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:24.186173 4431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 612964c3ba9140fea152adcd8e22048c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:24.187146 4431 raft_consensus.cc:504] T 00000000000000000000000000000000 P 612964c3ba9140fea152adcd8e22048c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "612964c3ba9140fea152adcd8e22048c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 41751 } }
I20260806 02:45:24.187464 4431 leader_election.cc:302] T 00000000000000000000000000000000 P 612964c3ba9140fea152adcd8e22048c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 612964c3ba9140fea152adcd8e22048c; no voters:
I20260806 02:45:24.187781 4431 leader_election.cc:288] T 00000000000000000000000000000000 P 612964c3ba9140fea152adcd8e22048c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:24.187916 4438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 612964c3ba9140fea152adcd8e22048c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:24.188211 4438 raft_consensus.cc:686] T 00000000000000000000000000000000 P 612964c3ba9140fea152adcd8e22048c [term 1 LEADER]: Becoming Leader. State: Replica: 612964c3ba9140fea152adcd8e22048c, State: Running, Role: LEADER
I20260806 02:45:24.188601 4438 consensus_queue.cc:237] T 00000000000000000000000000000000 P 612964c3ba9140fea152adcd8e22048c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "612964c3ba9140fea152adcd8e22048c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 41751 } }
I20260806 02:45:24.189203 4431 sys_catalog.cc:564] T 00000000000000000000000000000000 P 612964c3ba9140fea152adcd8e22048c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:24.190325 4439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 612964c3ba9140fea152adcd8e22048c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "612964c3ba9140fea152adcd8e22048c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "612964c3ba9140fea152adcd8e22048c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 41751 } } }
I20260806 02:45:24.190547 4439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 612964c3ba9140fea152adcd8e22048c [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:24.190652 4440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 612964c3ba9140fea152adcd8e22048c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 612964c3ba9140fea152adcd8e22048c. Latest consensus state: current_term: 1 leader_uuid: "612964c3ba9140fea152adcd8e22048c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "612964c3ba9140fea152adcd8e22048c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 41751 } } }
I20260806 02:45:24.190851 4440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 612964c3ba9140fea152adcd8e22048c [sys.catalog]: This master's current role is: LEADER
W20260806 02:45:24.194480 4456 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 612964c3ba9140fea152adcd8e22048c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:45:24.194572 4456 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:45:24.194669 4448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:24.195640 4448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:24.201967 4448 catalog_manager.cc:1347] Generated new cluster ID: 0f1c03ff4e8e4ea1880dc36babc67f53
I20260806 02:45:24.202066 4448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:24.211015 4448 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:24.212507 4448 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:24.231634 4448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 612964c3ba9140fea152adcd8e22048c: Generated new TSK 0
I20260806 02:45:24.232708 4448 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
I20260806 02:45:24.264047 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.129:0
--local_ip_for_outbound_sockets=127.1.183.129
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:41751
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:36719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:24.384773 4465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:24.385134 4465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:24.385231 4465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:24.389596 4465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:24.389706 4465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:24.389779 4465 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 02:45:24.389818 4465 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:45:24.389891 4465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.129
I20260806 02:45:24.396117 4465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:36719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.183.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.183.190:41751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.129
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:24.397459 4465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:24.398633 4465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:24.409121 4475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:24.409103 4480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:24.409096 4476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:24.409502 4465 server_base.cc:1034] running on GCE node
I20260806 02:45:24.409963 4465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:24.410719 4465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:24.411892 4465 hybrid_clock.cc:648] HybridClock initialized: now 1785984324411854 us; error 52 us; skew 500 ppm
Aug 06 02:45:24 dist-test-slave-gmjp krb5kdc[4294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984324, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:24.415095 4465 init.cc:377] Logged in from keytab as kudu/127.1.183.129@KRBTEST.COM (short username kudu)
I20260806 02:45:24.416111 4465 webserver.cc:460] Webserver started at http://127.1.183.129:40649/ using document root <none> and password file <none>
I20260806 02:45:24.416767 4465 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:24.416864 4465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:24.417105 4465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:24.419343 4465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-0/data/instance:
uuid: "5862084e3b914d4d877216c1c0ac63eb"
format_stamp: "Formatted at 2026-08-06 02:45:24 on dist-test-slave-gmjp"
server_key: "e82b6f73330b946c992df9b6d711e043"
server_key_iv: "aab6b75ba8bd3245868e00618bc298ec"
server_key_version: "encryptionkey@0"
I20260806 02:45:24.419975 4465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance:
uuid: "5862084e3b914d4d877216c1c0ac63eb"
format_stamp: "Formatted at 2026-08-06 02:45:24 on dist-test-slave-gmjp"
server_key: "e82b6f73330b946c992df9b6d711e043"
server_key_iv: "aab6b75ba8bd3245868e00618bc298ec"
server_key_version: "encryptionkey@0"
I20260806 02:45:24.424137 4465 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 02:45:24.426939 4491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:24.428393 4465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 02:45:24.428561 4465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-0/wal
uuid: "5862084e3b914d4d877216c1c0ac63eb"
format_stamp: "Formatted at 2026-08-06 02:45:24 on dist-test-slave-gmjp"
server_key: "e82b6f73330b946c992df9b6d711e043"
server_key_iv: "aab6b75ba8bd3245868e00618bc298ec"
server_key_version: "encryptionkey@0"
I20260806 02:45:24.428714 4465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:24.446406 4465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:24.449851 4465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:24.450098 4465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:24.451720 4465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:24.451804 4465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:24.451889 4465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:24.451931 4465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:45:24 dist-test-slave-gmjp krb5kdc[4294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984324, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:24.463732 4465 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.129:46263
I20260806 02:45:24.463811 4670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.129:46263 every 8 connection(s)
I20260806 02:45:24.464824 4465 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
I20260806 02:45:24.467872 4435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:24.452732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:44651 (local address 127.1.183.190:41751)
0806 02:45:24.452921 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:24.452927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:24.453919 (+ 992us) server_negotiation.cc:407] Connection header received
0806 02:45:24.455331 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:24.455337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:24.455413 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:24.455530 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:24.457687 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.458270 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.459581 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.459865 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.459993 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.463695 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:24.463701 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:24.463705 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:24.463745 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:24.465512 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:24.466084 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:24.466087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:24.466088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:24.466174 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:24.466571 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:24.466575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:24.466577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:24.466941 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:24.467060 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:24.467619 (+ 559us) server_negotiation.cc:299] Negotiation successful
0806 02:45:24.467744 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:45:24.468827 4503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:24.453144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:41751 (local address 127.1.183.129:44651)
0806 02:45:24.453694 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:24.453763 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:24.455026 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:24.455804 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:24.455817 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:24.456527 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:24.457414 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:24.457444 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.458445 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.458449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:24.459426 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:24.459437 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.460063 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.460068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:24.460324 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.461601 (+ 1277us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:24.461619 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:24.463455 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:24.465655 (+ 2200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:24.465661 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:24.465676 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:24.465996 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:24.466270 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:24.466272 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:24.466273 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:24.466427 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:24.467092 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:24.467103 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:24.467356 (+ 253us) client_negotiation.cc:769] Sending connection context
0806 02:45:24.467532 (+ 176us) client_negotiation.cc:240] Negotiation successful
0806 02:45:24.467816 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":124,"threads_started":1}
I20260806 02:45:24.470263 4672 heartbeater.cc:347] Connected to a master server at 127.1.183.190:41751
I20260806 02:45:24.470449 4672 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:24.471045 4672 heartbeater.cc:511] Master 127.1.183.190:41751 requested a full tablet report, sending...
I20260806 02:45:24.472625 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 4465
I20260806 02:45:24.472776 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance
I20260806 02:45:24.472914 4361 ts_manager.cc:194] Registered new tserver with Master: 5862084e3b914d4d877216c1c0ac63eb (127.1.183.129:46263)
I20260806 02:45:24.473073 1758 external_mini_cluster.cc:1428] Setting key c20145591921be46b307d39cfd3bca69
I20260806 02:45:24.474378 4361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.129@KRBTEST.COM'} at 127.1.183.129:44651
I20260806 02:45:24.481400 4435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:24.472686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:33395 (local address 127.1.183.190:41751)
0806 02:45:24.472799 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:24.472804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:24.473802 (+ 998us) server_negotiation.cc:407] Connection header received
0806 02:45:24.473946 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:24.473949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:24.473995 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:24.474070 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:24.474700 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.475379 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.476327 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.476558 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.476663 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.478119 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:24.478122 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:24.478125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:24.478148 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:24.479594 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:24.480130 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:24.480132 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:24.480134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:24.480254 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:24.480567 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:24.480570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:24.480571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:24.480746 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:24.480822 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:24.481171 (+ 349us) server_negotiation.cc:299] Negotiation successful
0806 02:45:24.481277 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 02:45:24.481643 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:24.472626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:41751 (local address 127.1.183.129:33395)
0806 02:45:24.473696 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:24.473714 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:24.473833 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:24.474117 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:24.474120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:24.474382 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:24.474532 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:24.474541 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.475511 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.475514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:24.476195 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:24.476206 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.476671 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.476675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:24.476796 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.477442 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:24.477462 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:24.477972 (+ 510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:24.479713 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:24.479719 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:24.479722 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:24.480029 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:24.480352 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:24.480357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:24.480362 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:24.480474 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:24.480848 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:24.480855 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:24.480990 (+ 135us) client_negotiation.cc:769] Sending connection context
0806 02:45:24.481125 (+ 135us) client_negotiation.cc:240] Negotiation successful
0806 02:45:24.481334 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":968,"mutex_wait_us":21,"thread_start_us":136,"threads_started":1}
WARNING: no policy specified for kudu/127.1.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
I20260806 02:45:24.503700 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.130:0
--local_ip_for_outbound_sockets=127.1.183.130
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:41751
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:36719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:24.621868 4683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:24.622213 4683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:24.622304 4683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:24.626447 4683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:24.626585 4683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:24.626664 4683 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 02:45:24.626708 4683 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:45:24.626782 4683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.130
I20260806 02:45:24.632516 4683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:36719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.183.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.183.190:41751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.130
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:24.633767 4683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:24.634831 4683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:24.644990 4688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:24.644990 4689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:24.645063 4683 server_base.cc:1034] running on GCE node
W20260806 02:45:24.645027 4691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:24.645815 4683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:24.646543 4683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:24.647717 4683 hybrid_clock.cc:648] HybridClock initialized: now 1785984324647696 us; error 34 us; skew 500 ppm
Aug 06 02:45:24 dist-test-slave-gmjp krb5kdc[4294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984324, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:24.650746 4683 init.cc:377] Logged in from keytab as kudu/127.1.183.130@KRBTEST.COM (short username kudu)
I20260806 02:45:24.651674 4683 webserver.cc:460] Webserver started at http://127.1.183.130:41561/ using document root <none> and password file <none>
I20260806 02:45:24.652323 4683 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:24.652415 4683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:24.652653 4683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:24.654783 4683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-1/data/instance:
uuid: "6642632003e04747a05e7229c96f4582"
format_stamp: "Formatted at 2026-08-06 02:45:24 on dist-test-slave-gmjp"
server_key: "1d7b350f8c450b67a81b3f339b71338b"
server_key_iv: "b9ce3e4ebab48e75ae2dceaec8d253a8"
server_key_version: "encryptionkey@0"
I20260806 02:45:24.655412 4683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance:
uuid: "6642632003e04747a05e7229c96f4582"
format_stamp: "Formatted at 2026-08-06 02:45:24 on dist-test-slave-gmjp"
server_key: "1d7b350f8c450b67a81b3f339b71338b"
server_key_iv: "b9ce3e4ebab48e75ae2dceaec8d253a8"
server_key_version: "encryptionkey@0"
I20260806 02:45:24.659332 4683 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:45:24.661864 4702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:24.663095 4683 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:45:24.663231 4683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-1/wal
uuid: "6642632003e04747a05e7229c96f4582"
format_stamp: "Formatted at 2026-08-06 02:45:24 on dist-test-slave-gmjp"
server_key: "1d7b350f8c450b67a81b3f339b71338b"
server_key_iv: "b9ce3e4ebab48e75ae2dceaec8d253a8"
server_key_version: "encryptionkey@0"
I20260806 02:45:24.663343 4683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:24.679979 4683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:24.683876 4683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:24.684119 4683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:24.685827 4683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:24.685902 4683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:24.685977 4683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:24.686017 4683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:45:24 dist-test-slave-gmjp krb5kdc[4294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984324, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:24.697898 4683 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.130:34573
I20260806 02:45:24.697921 4874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.130:34573 every 8 connection(s)
I20260806 02:45:24.699004 4683 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
I20260806 02:45:24.700956 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 4683
I20260806 02:45:24.701076 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance
I20260806 02:45:24.701359 1758 external_mini_cluster.cc:1428] Setting key 37511f25a66f214d82311519b15b19a1
I20260806 02:45:24.702262 4435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:24.686730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:60125 (local address 127.1.183.190:41751)
0806 02:45:24.686914 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:24.686922 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:24.687754 (+ 832us) server_negotiation.cc:407] Connection header received
0806 02:45:24.688901 (+ 1147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:24.688907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:24.688973 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:24.689057 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:24.690833 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.691809 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.693583 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.693893 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.694013 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.697956 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:24.697960 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:24.697963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:24.698001 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:24.699844 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:24.700455 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:24.700459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:24.700461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:24.700546 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:24.700991 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:24.700994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:24.700996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:24.701248 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:24.701338 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:24.701975 (+ 637us) server_negotiation.cc:299] Negotiation successful
0806 02:45:24.702091 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:45:24.703315 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:24.687131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:41751 (local address 127.1.183.130:60125)
0806 02:45:24.687584 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:24.687637 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:24.688627 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:24.689448 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:24.689454 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:24.689973 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:24.690667 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:24.690687 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.691950 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.691954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:24.693462 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:24.693475 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.694072 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.694076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:24.694304 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.695730 (+ 1426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:24.695754 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:24.697742 (+ 1988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:24.699960 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:24.699966 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:24.699980 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:24.700343 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:24.700640 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:24.700643 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:24.700645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:24.700855 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:24.701362 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:24.701370 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:24.701680 (+ 310us) client_negotiation.cc:769] Sending connection context
0806 02:45:24.701871 (+ 191us) client_negotiation.cc:240] Negotiation successful
0806 02:45:24.702236 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":129,"threads_started":1}
I20260806 02:45:24.704710 4876 heartbeater.cc:347] Connected to a master server at 127.1.183.190:41751
I20260806 02:45:24.704890 4876 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:24.705437 4876 heartbeater.cc:511] Master 127.1.183.190:41751 requested a full tablet report, sending...
I20260806 02:45:24.706799 4362 ts_manager.cc:194] Registered new tserver with Master: 6642632003e04747a05e7229c96f4582 (127.1.183.130:34573)
I20260806 02:45:24.707713 4362 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.130@KRBTEST.COM'} at 127.1.183.130:60125
WARNING: no policy specified for kudu/127.1.183.131@KRBTEST.COM; defaulting to no policy
I20260806 02:45:24.716591 4890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:24.707474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:41751 (local address 127.1.183.130:34107)
0806 02:45:24.707819 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:24.707840 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:24.707954 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:24.708564 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:24.708567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:24.708838 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:24.709016 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:24.709025 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.710183 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.710186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:24.710894 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:24.710904 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.711676 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.711679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:24.711794 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.712454 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:24.712466 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:24.712833 (+ 367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:24.714888 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:24.714891 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:24.714893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:24.715122 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:24.715440 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:24.715443 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:24.715444 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:24.715524 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:24.715935 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:24.715938 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:24.716049 (+ 111us) client_negotiation.cc:769] Sending connection context
0806 02:45:24.716194 (+ 145us) client_negotiation.cc:240] Negotiation successful
0806 02:45:24.716371 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":30,"thread_start_us":109,"threads_started":1}
I20260806 02:45:24.716593 4435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:24.708087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:34107 (local address 127.1.183.190:41751)
0806 02:45:24.708292 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:24.708300 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:24.708318 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:45:24.708363 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:24.708369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:24.708426 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:24.708519 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:24.709174 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.709990 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.711042 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.711499 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.711635 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.713003 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:24.713012 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:24.713017 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:24.713053 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:24.714748 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:24.715228 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:24.715234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:24.715235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:24.715327 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:24.715587 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:24.715589 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:24.715590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:24.715822 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:24.715945 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:24.716327 (+ 382us) server_negotiation.cc:299] Negotiation successful
0806 02:45:24.716462 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
I20260806 02:45:24.736507 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.131:0
--local_ip_for_outbound_sockets=127.1.183.131
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:41751
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:36719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:24.867838 4893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:24.868136 4893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:24.868257 4893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:24.872725 4893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:24.872813 4893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:24.872862 4893 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 02:45:24.872882 4893 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:45:24.872925 4893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.131
I20260806 02:45:24.879048 4893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:36719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.183.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.183.190:41751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.131
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:24.880138 4893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:24.881240 4893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:24.892748 4903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:24.892799 4902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:24.892786 4906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:24.893162 4893 server_base.cc:1034] running on GCE node
I20260806 02:45:24.893646 4893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:24.894377 4893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:24.895556 4893 hybrid_clock.cc:648] HybridClock initialized: now 1785984324895538 us; error 29 us; skew 500 ppm
Aug 06 02:45:24 dist-test-slave-gmjp krb5kdc[4294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984324, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:24.898875 4893 init.cc:377] Logged in from keytab as kudu/127.1.183.131@KRBTEST.COM (short username kudu)
I20260806 02:45:24.899969 4893 webserver.cc:460] Webserver started at http://127.1.183.131:44151/ using document root <none> and password file <none>
I20260806 02:45:24.900703 4893 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:24.900821 4893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:24.901088 4893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:24.903439 4893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-2/data/instance:
uuid: "6ff00848052f43e4b76c6b9c02e26f02"
format_stamp: "Formatted at 2026-08-06 02:45:24 on dist-test-slave-gmjp"
server_key: "261a3f144f83588d839dffe003e1d8c3"
server_key_iv: "b10922157072fcaef5097408db76c591"
server_key_version: "encryptionkey@0"
I20260806 02:45:24.904111 4893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance:
uuid: "6ff00848052f43e4b76c6b9c02e26f02"
format_stamp: "Formatted at 2026-08-06 02:45:24 on dist-test-slave-gmjp"
server_key: "261a3f144f83588d839dffe003e1d8c3"
server_key_iv: "b10922157072fcaef5097408db76c591"
server_key_version: "encryptionkey@0"
I20260806 02:45:24.908741 4893 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:45:24.911583 4915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:24.913086 4893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:24.913215 4893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-2/wal
uuid: "6ff00848052f43e4b76c6b9c02e26f02"
format_stamp: "Formatted at 2026-08-06 02:45:24 on dist-test-slave-gmjp"
server_key: "261a3f144f83588d839dffe003e1d8c3"
server_key_iv: "b10922157072fcaef5097408db76c591"
server_key_version: "encryptionkey@0"
I20260806 02:45:24.913321 4893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:24.925069 4893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:24.928396 4893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:24.928570 4893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:24.930078 4893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:24.930132 4893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:24.930173 4893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:24.930238 4893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:45:24 dist-test-slave-gmjp krb5kdc[4294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984324, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:24.941613 4893 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.131:37335
I20260806 02:45:24.941685 5106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.131:37335 every 8 connection(s)
I20260806 02:45:24.942646 4893 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
I20260806 02:45:24.945168 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 4893
I20260806 02:45:24.945317 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance
I20260806 02:45:24.945602 1758 external_mini_cluster.cc:1428] Setting key 0c30153e65a972a7a9b7d5ca29cbf2e9
I20260806 02:45:24.946178 4435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:24.931018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:59931 (local address 127.1.183.190:41751)
0806 02:45:24.931150 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:24.931154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:24.932300 (+ 1146us) server_negotiation.cc:407] Connection header received
0806 02:45:24.933628 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:24.933635 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:24.933719 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:24.933839 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:24.935567 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.936477 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.938008 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.938438 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.938583 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.941844 (+ 3261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:24.941848 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:24.941851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:24.941888 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:24.943634 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:24.944398 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:24.944401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:24.944403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:24.944499 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:24.944883 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:24.944886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:24.944888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:24.945096 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:24.945208 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:24.945842 (+ 634us) server_negotiation.cc:299] Negotiation successful
0806 02:45:24.946024 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:45:24.947175 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:24.931610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:41751 (local address 127.1.183.131:59931)
0806 02:45:24.932103 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:24.932147 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:24.933341 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:24.934039 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:24.934051 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:24.934532 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:24.935334 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:24.935354 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.936618 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.936621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:24.937802 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:24.937813 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.938627 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.938632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:24.938859 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.939570 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:24.939596 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:24.941642 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:24.943769 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:24.943775 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:24.943792 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:24.944285 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:24.944601 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:24.944604 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:24.944606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:24.944776 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:24.945207 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:24.945216 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:24.945551 (+ 335us) client_negotiation.cc:769] Sending connection context
0806 02:45:24.945743 (+ 192us) client_negotiation.cc:240] Negotiation successful
0806 02:45:24.946116 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":132,"threads_started":1}
I20260806 02:45:24.948526 5107 heartbeater.cc:347] Connected to a master server at 127.1.183.190:41751
I20260806 02:45:24.948695 5107 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:24.949256 5107 heartbeater.cc:511] Master 127.1.183.190:41751 requested a full tablet report, sending...
I20260806 02:45:24.950471 4361 ts_manager.cc:194] Registered new tserver with Master: 6ff00848052f43e4b76c6b9c02e26f02 (127.1.183.131:37335)
I20260806 02:45:24.951252 4361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.131@KRBTEST.COM'} at 127.1.183.131:59931
I20260806 02:45:24.958463 4435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:24.950972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:35313 (local address 127.1.183.190:41751)
0806 02:45:24.951108 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:24.951112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:24.951308 (+ 196us) server_negotiation.cc:407] Connection header received
0806 02:45:24.951449 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:24.951452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:24.951498 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:24.951589 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:24.952361 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.953104 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.953976 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.954183 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.954262 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.955343 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:24.955346 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:24.955349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:24.955371 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:24.956846 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:24.957349 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:24.957351 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:24.957352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:24.957428 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:24.957703 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:24.957705 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:24.957706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:24.957880 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:24.957973 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:24.958224 (+ 251us) server_negotiation.cc:299] Negotiation successful
0806 02:45:24.958323 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:45:24.958587 5118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:24.950899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:41751 (local address 127.1.183.131:35313)
0806 02:45:24.951221 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:24.951234 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:24.951323 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:24.951872 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:24.951874 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:24.952073 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:24.952235 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:24.952243 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.953259 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.953262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:24.953863 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:24.953872 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.954296 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.954299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:24.954381 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.954870 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:24.954882 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:24.955213 (+ 331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:24.956983 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:24.956986 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:24.956997 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:24.957233 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:24.957525 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:24.957527 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:24.957529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:24.957618 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:24.957982 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:24.957985 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:24.958072 (+ 87us) client_negotiation.cc:769] Sending connection context
0806 02:45:24.958165 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 02:45:24.958376 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":98,"threads_started":1}
I20260806 02:45:24.960319 1758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:45:24 dist-test-slave-gmjp krb5kdc[4294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:45:24 dist-test-slave-gmjp krb5kdc[4294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:24.990381 4435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:24.981542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59580 (local address 127.1.183.190:41751)
0806 02:45:24.981680 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:24.981684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:24.981695 (+ 11us) server_negotiation.cc:407] Connection header received
0806 02:45:24.981737 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:24.981740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:24.981779 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:24.981876 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:24.982653 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.983182 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.984073 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:24.984359 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:24.984476 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:24.987117 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:24.987121 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:24.987123 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:24.987153 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:24.988690 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:24.989093 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:24.989096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:24.989097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:24.989176 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:24.989447 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:24.989468 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:24.989473 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:24.989719 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:24.989867 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:24.990145 (+ 278us) server_negotiation.cc:299] Negotiation successful
0806 02:45:24.990251 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:45:24.993862 4362 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:45:24.997020 4362 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:45:25.013357 5139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:25.007138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:46263 (local address 127.0.0.1:45856)
0806 02:45:25.007803 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:25.007820 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:25.007912 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:25.009061 (+ 1149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:25.009064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:25.009083 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:25.009188 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:25.009193 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.010528 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.010531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:25.011606 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:25.011615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.012677 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.012681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:25.012851 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:25.012902 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 02:45:25.013036 (+ 134us) client_negotiation.cc:240] Negotiation successful
0806 02:45:25.013178 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":7,"thread_start_us":102,"threads_started":1}
I20260806 02:45:25.013528 5137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:25.006296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:34573 (local address 127.0.0.1:50558)
0806 02:45:25.006743 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:25.006804 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:25.006943 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:25.008759 (+ 1816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:25.008763 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:25.008792 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:25.008902 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:25.008913 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.009983 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.009990 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:25.011877 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:25.011890 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.013022 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.013026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:25.013115 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:25.013139 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 02:45:25.013231 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 02:45:25.013313 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":225,"thread_start_us":118,"threads_started":1}
I20260806 02:45:25.013566 5138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:25.006474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50558 (local address 127.1.183.130:34573)
0806 02:45:25.008261 (+ 1787us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:25.008276 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:25.008312 (+ 36us) server_negotiation.cc:407] Connection header received
0806 02:45:25.008388 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:25.008393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:25.008600 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:25.008786 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:25.009048 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.009853 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.012025 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.012889 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.013022 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:25.013158 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:25.013286 (+ 128us) server_negotiation.cc:299] Negotiation successful
0806 02:45:25.013394 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1596,"thread_start_us":123,"threads_started":1}
I20260806 02:45:25.013988 5143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:25.007316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45856 (local address 127.1.183.129:46263)
0806 02:45:25.008606 (+ 1290us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:25.008614 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:25.008638 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:45:25.008712 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:25.008719 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:25.008893 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:25.009330 (+ 437us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:25.009470 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.010392 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.011742 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.012541 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.012656 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:25.012812 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:25.013702 (+ 890us) server_negotiation.cc:299] Negotiation successful
0806 02:45:25.013813 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1177,"thread_start_us":102,"threads_started":1}
I20260806 02:45:25.014430 5141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:25.006626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:37335 (local address 127.0.0.1:46356)
0806 02:45:25.007384 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:25.007399 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:25.007498 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:25.008307 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:25.008313 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:25.008343 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:25.008517 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:25.008524 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.010030 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.010033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:25.011880 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:25.011890 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.013328 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.013332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:25.014121 (+ 789us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:25.014148 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 02:45:25.014230 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 02:45:25.014303 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":117,"thread_start_us":92,"threads_started":1}
I20260806 02:45:25.014950 5142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:25.007200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46356 (local address 127.1.183.131:37335)
0806 02:45:25.007730 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:25.007745 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:25.007767 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:45:25.007856 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:25.007863 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:25.008101 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:25.008322 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:25.008658 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.009849 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.012295 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.013013 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.013114 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:25.013262 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:25.014702 (+ 1440us) server_negotiation.cc:299] Negotiation successful
0806 02:45:25.014807 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":209,"threads_started":1}
I20260806 02:45:25.016791 4562 tablet_service.cc:1467] Processing CreateTablet for tablet b1e292126bf64650af110300e1c420de (DEFAULT_TABLE table=test-table [id=028f0b220bbc47c39080e1875dc0504b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:25.016791 4781 tablet_service.cc:1467] Processing CreateTablet for tablet b1e292126bf64650af110300e1c420de (DEFAULT_TABLE table=test-table [id=028f0b220bbc47c39080e1875dc0504b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:25.017925 4562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1e292126bf64650af110300e1c420de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:25.017925 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1e292126bf64650af110300e1c420de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:25.017570 5004 tablet_service.cc:1467] Processing CreateTablet for tablet b1e292126bf64650af110300e1c420de (DEFAULT_TABLE table=test-table [id=028f0b220bbc47c39080e1875dc0504b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:25.018486 5004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1e292126bf64650af110300e1c420de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:25.024310 5144 tablet_bootstrap.cc:492] T b1e292126bf64650af110300e1c420de P 5862084e3b914d4d877216c1c0ac63eb: Bootstrap starting.
I20260806 02:45:25.025921 5145 tablet_bootstrap.cc:492] T b1e292126bf64650af110300e1c420de P 6642632003e04747a05e7229c96f4582: Bootstrap starting.
I20260806 02:45:25.026952 5148 tablet_bootstrap.cc:492] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02: Bootstrap starting.
I20260806 02:45:25.027211 5144 tablet_bootstrap.cc:654] T b1e292126bf64650af110300e1c420de P 5862084e3b914d4d877216c1c0ac63eb: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:25.028184 5144 log.cc:824] T b1e292126bf64650af110300e1c420de P 5862084e3b914d4d877216c1c0ac63eb: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:25.028501 5145 tablet_bootstrap.cc:654] T b1e292126bf64650af110300e1c420de P 6642632003e04747a05e7229c96f4582: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:25.029052 5148 tablet_bootstrap.cc:654] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:25.029404 5145 log.cc:824] T b1e292126bf64650af110300e1c420de P 6642632003e04747a05e7229c96f4582: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:25.029891 5148 log.cc:824] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:25.030887 5144 tablet_bootstrap.cc:492] T b1e292126bf64650af110300e1c420de P 5862084e3b914d4d877216c1c0ac63eb: No bootstrap required, opened a new log
I20260806 02:45:25.031108 5144 ts_tablet_manager.cc:1397] T b1e292126bf64650af110300e1c420de P 5862084e3b914d4d877216c1c0ac63eb: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 02:45:25.031553 5145 tablet_bootstrap.cc:492] T b1e292126bf64650af110300e1c420de P 6642632003e04747a05e7229c96f4582: No bootstrap required, opened a new log
I20260806 02:45:25.031754 5145 ts_tablet_manager.cc:1397] T b1e292126bf64650af110300e1c420de P 6642632003e04747a05e7229c96f4582: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:45:25.031787 5148 tablet_bootstrap.cc:492] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02: No bootstrap required, opened a new log
I20260806 02:45:25.032039 5148 ts_tablet_manager.cc:1397] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:45:25.035112 5148 raft_consensus.cc:348] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:25.035416 5148 raft_consensus.cc:374] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:25.035472 5148 raft_consensus.cc:729] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff00848052f43e4b76c6b9c02e26f02, State: Initialized, Role: FOLLOWER
I20260806 02:45:25.035892 5144 raft_consensus.cc:348] T b1e292126bf64650af110300e1c420de P 5862084e3b914d4d877216c1c0ac63eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:25.035861 5145 raft_consensus.cc:348] T b1e292126bf64650af110300e1c420de P 6642632003e04747a05e7229c96f4582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:25.036038 5148 consensus_queue.cc:260] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:25.036135 5144 raft_consensus.cc:374] T b1e292126bf64650af110300e1c420de P 5862084e3b914d4d877216c1c0ac63eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:25.036135 5145 raft_consensus.cc:374] T b1e292126bf64650af110300e1c420de P 6642632003e04747a05e7229c96f4582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:25.036230 5144 raft_consensus.cc:729] T b1e292126bf64650af110300e1c420de P 5862084e3b914d4d877216c1c0ac63eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5862084e3b914d4d877216c1c0ac63eb, State: Initialized, Role: FOLLOWER
I20260806 02:45:25.036230 5145 raft_consensus.cc:729] T b1e292126bf64650af110300e1c420de P 6642632003e04747a05e7229c96f4582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6642632003e04747a05e7229c96f4582, State: Initialized, Role: FOLLOWER
I20260806 02:45:25.036712 5145 consensus_queue.cc:260] T b1e292126bf64650af110300e1c420de P 6642632003e04747a05e7229c96f4582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:25.036712 5144 consensus_queue.cc:260] T b1e292126bf64650af110300e1c420de P 5862084e3b914d4d877216c1c0ac63eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:25.037328 5148 ts_tablet_manager.cc:1428] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 02:45:25.037386 5107 heartbeater.cc:503] Master 127.1.183.190:41751 was elected leader, sending a full tablet report...
I20260806 02:45:25.037722 5144 ts_tablet_manager.cc:1428] T b1e292126bf64650af110300e1c420de P 5862084e3b914d4d877216c1c0ac63eb: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260806 02:45:25.038074 4672 heartbeater.cc:503] Master 127.1.183.190:41751 was elected leader, sending a full tablet report...
I20260806 02:45:25.038623 5145 ts_tablet_manager.cc:1428] T b1e292126bf64650af110300e1c420de P 6642632003e04747a05e7229c96f4582: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 02:45:25.038950 4876 heartbeater.cc:503] Master 127.1.183.190:41751 was elected leader, sending a full tablet report...
W20260806 02:45:25.193781 5111 tablet.cc:2369] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:45:25.200410 4877 tablet.cc:2369] T b1e292126bf64650af110300e1c420de P 6642632003e04747a05e7229c96f4582: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:45:25.216512 4673 tablet.cc:2369] T b1e292126bf64650af110300e1c420de P 5862084e3b914d4d877216c1c0ac63eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:45:25.365073 5154 raft_consensus.cc:482] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:45:25.365307 5154 raft_consensus.cc:504] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:25.366595 5154 leader_election.cc:288] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6642632003e04747a05e7229c96f4582 (127.1.183.130:34573), 5862084e3b914d4d877216c1c0ac63eb (127.1.183.129:46263)
I20260806 02:45:25.371272 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:25.366969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:34573 (local address 127.1.183.131:42615)
0806 02:45:25.367130 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:25.367168 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:25.367291 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:25.367657 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:25.367661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:25.367685 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:25.367838 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:25.367845 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.368803 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.368807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:25.370056 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:25.370071 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.370803 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.370808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:25.370923 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:25.370949 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 02:45:25.371028 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 02:45:25.371102 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260806 02:45:25.371416 5138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:25.367107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:42615 (local address 127.1.183.130:34573)
0806 02:45:25.367258 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:25.367263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:25.367279 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:45:25.367443 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:25.367447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:25.367497 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:25.367590 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:25.367959 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.368627 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.370204 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.370667 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.370794 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:25.370858 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:25.371085 (+ 227us) server_negotiation.cc:299] Negotiation successful
0806 02:45:25.371224 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:45:25.372515 5158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:25.367315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:46263 (local address 127.1.183.131:41623)
0806 02:45:25.367687 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:25.367701 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:25.367831 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:25.368134 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:25.368136 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:25.368199 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:25.368836 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:25.368850 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.369885 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.369889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:25.371144 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:25.371154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.372052 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.372055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:25.372125 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:25.372140 (+ 15us) client_negotiation.cc:769] Sending connection context
0806 02:45:25.372253 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 02:45:25.372352 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":138,"threads_started":1}
I20260806 02:45:25.372611 5143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:25.367478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:41623 (local address 127.1.183.129:46263)
0806 02:45:25.367632 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:25.367637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:25.367785 (+ 148us) server_negotiation.cc:407] Connection header received
0806 02:45:25.367948 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:25.367950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:25.368002 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:25.368103 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:25.368982 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.369753 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.371282 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.371843 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.371922 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:25.371959 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:25.372307 (+ 348us) server_negotiation.cc:299] Negotiation successful
0806 02:45:25.372459 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:45:25.373567 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e292126bf64650af110300e1c420de" candidate_uuid: "6ff00848052f43e4b76c6b9c02e26f02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6642632003e04747a05e7229c96f4582" is_pre_election: true
I20260806 02:45:25.373533 4600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e292126bf64650af110300e1c420de" candidate_uuid: "6ff00848052f43e4b76c6b9c02e26f02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5862084e3b914d4d877216c1c0ac63eb" is_pre_election: true
I20260806 02:45:25.373960 4808 raft_consensus.cc:2454] T b1e292126bf64650af110300e1c420de P 6642632003e04747a05e7229c96f4582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ff00848052f43e4b76c6b9c02e26f02 in term 0.
I20260806 02:45:25.374003 4600 raft_consensus.cc:2454] T b1e292126bf64650af110300e1c420de P 5862084e3b914d4d877216c1c0ac63eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ff00848052f43e4b76c6b9c02e26f02 in term 0.
I20260806 02:45:25.374630 4919 leader_election.cc:302] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6642632003e04747a05e7229c96f4582, 6ff00848052f43e4b76c6b9c02e26f02; no voters:
I20260806 02:45:25.375033 5154 raft_consensus.cc:2781] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:45:25.375124 5154 raft_consensus.cc:482] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:45:25.375156 5154 raft_consensus.cc:3037] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:25.376451 5154 raft_consensus.cc:504] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:25.376844 5154 leader_election.cc:288] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02 [CANDIDATE]: Term 1 election: Requested vote from peers 6642632003e04747a05e7229c96f4582 (127.1.183.130:34573), 5862084e3b914d4d877216c1c0ac63eb (127.1.183.129:46263)
I20260806 02:45:25.377280 4600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e292126bf64650af110300e1c420de" candidate_uuid: "6ff00848052f43e4b76c6b9c02e26f02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5862084e3b914d4d877216c1c0ac63eb"
I20260806 02:45:25.377279 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e292126bf64650af110300e1c420de" candidate_uuid: "6ff00848052f43e4b76c6b9c02e26f02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6642632003e04747a05e7229c96f4582"
I20260806 02:45:25.377436 4808 raft_consensus.cc:3037] T b1e292126bf64650af110300e1c420de P 6642632003e04747a05e7229c96f4582 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:25.377424 4600 raft_consensus.cc:3037] T b1e292126bf64650af110300e1c420de P 5862084e3b914d4d877216c1c0ac63eb [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:25.378540 4600 raft_consensus.cc:2454] T b1e292126bf64650af110300e1c420de P 5862084e3b914d4d877216c1c0ac63eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ff00848052f43e4b76c6b9c02e26f02 in term 1.
I20260806 02:45:25.378542 4808 raft_consensus.cc:2454] T b1e292126bf64650af110300e1c420de P 6642632003e04747a05e7229c96f4582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ff00848052f43e4b76c6b9c02e26f02 in term 1.
I20260806 02:45:25.378877 4919 leader_election.cc:302] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6642632003e04747a05e7229c96f4582, 6ff00848052f43e4b76c6b9c02e26f02; no voters:
I20260806 02:45:25.379139 5154 raft_consensus.cc:2781] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:25.379372 5154 raft_consensus.cc:686] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02 [term 1 LEADER]: Becoming Leader. State: Replica: 6ff00848052f43e4b76c6b9c02e26f02, State: Running, Role: LEADER
I20260806 02:45:25.379729 5154 consensus_queue.cc:237] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:25.383497 4361 catalog_manager.cc:5495] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ff00848052f43e4b76c6b9c02e26f02 (127.1.183.131). New cstate: current_term: 1 leader_uuid: "6ff00848052f43e4b76c6b9c02e26f02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } health_report { overall_health: HEALTHY } } }
Aug 06 02:45:25 dist-test-slave-gmjp krb5kdc[4294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984324, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:25.486876 4435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:25.478106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59588 (local address 127.1.183.190:41751)
0806 02:45:25.478234 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:25.478238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:25.478622 (+ 384us) server_negotiation.cc:407] Connection header received
0806 02:45:25.478733 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:25.478735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:25.478780 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:25.478848 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:25.479613 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.480133 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.480948 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.481153 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.481231 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:25.483669 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:25.483672 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:25.483674 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:25.483696 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:25.485241 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:25.485694 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:25.485697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:25.485698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:25.485768 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:25.486012 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:25.486014 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:25.486015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:25.486186 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:25.486262 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:25.486594 (+ 332us) server_negotiation.cc:299] Negotiation successful
0806 02:45:25.486731 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":45}
I20260806 02:45:25.486879 5171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:25.478066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:41751 (local address 127.0.0.1:59588)
0806 02:45:25.478524 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:25.478537 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:25.478648 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:25.478886 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:25.478888 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:25.479354 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:25.479503 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:25.479509 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.480282 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.480285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:25.480843 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:25.480850 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.481287 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.481291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:25.481360 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:25.481992 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:25.482005 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:25.483522 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:25.485342 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:25.485352 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:25.485357 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:25.485607 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:25.485845 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:25.485847 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:25.485848 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:25.485934 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:25.486314 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:25.486320 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:25.486433 (+ 113us) client_negotiation.cc:769] Sending connection context
0806 02:45:25.486539 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 02:45:25.486673 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":226,"threads_started":1}
I20260806 02:45:25.491379 4361 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:59588:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 02:45:25.492126 4361 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:45:25.497288 4781 tablet_service.cc:1467] Processing CreateTablet for tablet 196e14e9ce3d4d4091be80aaa440e611 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5f7e8455ff9d4149bcaa9206c705a5a7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:45:25.497359 5004 tablet_service.cc:1467] Processing CreateTablet for tablet 196e14e9ce3d4d4091be80aaa440e611 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5f7e8455ff9d4149bcaa9206c705a5a7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:45:25.497191 4562 tablet_service.cc:1467] Processing CreateTablet for tablet 196e14e9ce3d4d4091be80aaa440e611 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5f7e8455ff9d4149bcaa9206c705a5a7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:45:25.497916 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 196e14e9ce3d4d4091be80aaa440e611. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:25.497969 5004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 196e14e9ce3d4d4091be80aaa440e611. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:25.497982 4562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 196e14e9ce3d4d4091be80aaa440e611. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:25.500572 5145 tablet_bootstrap.cc:492] T 196e14e9ce3d4d4091be80aaa440e611 P 6642632003e04747a05e7229c96f4582: Bootstrap starting.
I20260806 02:45:25.501154 5144 tablet_bootstrap.cc:492] T 196e14e9ce3d4d4091be80aaa440e611 P 5862084e3b914d4d877216c1c0ac63eb: Bootstrap starting.
I20260806 02:45:25.501551 5145 tablet_bootstrap.cc:654] T 196e14e9ce3d4d4091be80aaa440e611 P 6642632003e04747a05e7229c96f4582: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:25.501909 5144 tablet_bootstrap.cc:654] T 196e14e9ce3d4d4091be80aaa440e611 P 5862084e3b914d4d877216c1c0ac63eb: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:25.502837 5145 tablet_bootstrap.cc:492] T 196e14e9ce3d4d4091be80aaa440e611 P 6642632003e04747a05e7229c96f4582: No bootstrap required, opened a new log
I20260806 02:45:25.502950 5145 ts_tablet_manager.cc:1397] T 196e14e9ce3d4d4091be80aaa440e611 P 6642632003e04747a05e7229c96f4582: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:45:25.503153 5144 tablet_bootstrap.cc:492] T 196e14e9ce3d4d4091be80aaa440e611 P 5862084e3b914d4d877216c1c0ac63eb: No bootstrap required, opened a new log
I20260806 02:45:25.503243 5144 ts_tablet_manager.cc:1397] T 196e14e9ce3d4d4091be80aaa440e611 P 5862084e3b914d4d877216c1c0ac63eb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 02:45:25.503365 5148 tablet_bootstrap.cc:492] T 196e14e9ce3d4d4091be80aaa440e611 P 6ff00848052f43e4b76c6b9c02e26f02: Bootstrap starting.
I20260806 02:45:25.503494 5145 raft_consensus.cc:348] T 196e14e9ce3d4d4091be80aaa440e611 P 6642632003e04747a05e7229c96f4582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:25.503625 5145 raft_consensus.cc:374] T 196e14e9ce3d4d4091be80aaa440e611 P 6642632003e04747a05e7229c96f4582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:25.503667 5145 raft_consensus.cc:729] T 196e14e9ce3d4d4091be80aaa440e611 P 6642632003e04747a05e7229c96f4582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6642632003e04747a05e7229c96f4582, State: Initialized, Role: FOLLOWER
I20260806 02:45:25.503746 5144 raft_consensus.cc:348] T 196e14e9ce3d4d4091be80aaa440e611 P 5862084e3b914d4d877216c1c0ac63eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:25.503803 5145 consensus_queue.cc:260] T 196e14e9ce3d4d4091be80aaa440e611 P 6642632003e04747a05e7229c96f4582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:25.503861 5144 raft_consensus.cc:374] T 196e14e9ce3d4d4091be80aaa440e611 P 5862084e3b914d4d877216c1c0ac63eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:25.503897 5144 raft_consensus.cc:729] T 196e14e9ce3d4d4091be80aaa440e611 P 5862084e3b914d4d877216c1c0ac63eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5862084e3b914d4d877216c1c0ac63eb, State: Initialized, Role: FOLLOWER
I20260806 02:45:25.504024 5144 consensus_queue.cc:260] T 196e14e9ce3d4d4091be80aaa440e611 P 5862084e3b914d4d877216c1c0ac63eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:25.504140 5155 tablet_replica.cc:439] T 196e14e9ce3d4d4091be80aaa440e611 P 6642632003e04747a05e7229c96f4582: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } } }
I20260806 02:45:25.504311 5144 ts_tablet_manager.cc:1428] T 196e14e9ce3d4d4091be80aaa440e611 P 5862084e3b914d4d877216c1c0ac63eb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:45:25.504356 5155 tablet_replica.cc:442] T 196e14e9ce3d4d4091be80aaa440e611 P 6642632003e04747a05e7229c96f4582: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:25.504522 5148 tablet_bootstrap.cc:654] T 196e14e9ce3d4d4091be80aaa440e611 P 6ff00848052f43e4b76c6b9c02e26f02: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:25.504312 5157 tablet_replica.cc:439] T 196e14e9ce3d4d4091be80aaa440e611 P 5862084e3b914d4d877216c1c0ac63eb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } } }
I20260806 02:45:25.504915 5157 tablet_replica.cc:442] T 196e14e9ce3d4d4091be80aaa440e611 P 5862084e3b914d4d877216c1c0ac63eb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:25.506031 5148 tablet_bootstrap.cc:492] T 196e14e9ce3d4d4091be80aaa440e611 P 6ff00848052f43e4b76c6b9c02e26f02: No bootstrap required, opened a new log
I20260806 02:45:25.506150 5148 ts_tablet_manager.cc:1397] T 196e14e9ce3d4d4091be80aaa440e611 P 6ff00848052f43e4b76c6b9c02e26f02: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 02:45:25.506619 5148 raft_consensus.cc:348] T 196e14e9ce3d4d4091be80aaa440e611 P 6ff00848052f43e4b76c6b9c02e26f02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:25.506783 5148 raft_consensus.cc:374] T 196e14e9ce3d4d4091be80aaa440e611 P 6ff00848052f43e4b76c6b9c02e26f02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:25.506824 5148 raft_consensus.cc:729] T 196e14e9ce3d4d4091be80aaa440e611 P 6ff00848052f43e4b76c6b9c02e26f02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff00848052f43e4b76c6b9c02e26f02, State: Initialized, Role: FOLLOWER
I20260806 02:45:25.506961 5148 consensus_queue.cc:260] T 196e14e9ce3d4d4091be80aaa440e611 P 6ff00848052f43e4b76c6b9c02e26f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:25.507289 5148 ts_tablet_manager.cc:1428] T 196e14e9ce3d4d4091be80aaa440e611 P 6ff00848052f43e4b76c6b9c02e26f02: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 02:45:25.507246 5154 tablet_replica.cc:439] T 196e14e9ce3d4d4091be80aaa440e611 P 6ff00848052f43e4b76c6b9c02e26f02: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } } }
I20260806 02:45:25.507391 5154 tablet_replica.cc:442] T 196e14e9ce3d4d4091be80aaa440e611 P 6ff00848052f43e4b76c6b9c02e26f02: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:25.507066 5145 ts_tablet_manager.cc:1428] T 196e14e9ce3d4d4091be80aaa440e611 P 6642632003e04747a05e7229c96f4582: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260806 02:45:25.570492 5155 raft_consensus.cc:482] T 196e14e9ce3d4d4091be80aaa440e611 P 6642632003e04747a05e7229c96f4582 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:45:25.570636 5155 raft_consensus.cc:504] T 196e14e9ce3d4d4091be80aaa440e611 P 6642632003e04747a05e7229c96f4582 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:25.571970 5155 leader_election.cc:288] T 196e14e9ce3d4d4091be80aaa440e611 P 6642632003e04747a05e7229c96f4582 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5862084e3b914d4d877216c1c0ac63eb (127.1.183.129:46263), 6ff00848052f43e4b76c6b9c02e26f02 (127.1.183.131:37335)
I20260806 02:45:25.576048 5143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:25.572305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:54263 (local address 127.1.183.129:46263)
0806 02:45:25.572473 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:25.572479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:25.572662 (+ 183us) server_negotiation.cc:407] Connection header received
0806 02:45:25.572838 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:25.572841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:25.572890 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:25.572975 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:25.573541 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.574046 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.575064 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.575478 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.575606 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:25.575660 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:25.575845 (+ 185us) server_negotiation.cc:299] Negotiation successful
0806 02:45:25.575916 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 02:45:25.576048 5181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:25.572401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:34665 (local address 127.1.183.131:37335)
0806 02:45:25.572623 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:25.572643 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:25.572685 (+ 42us) server_negotiation.cc:407] Connection header received
0806 02:45:25.572862 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:25.572865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:25.572915 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:25.573009 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:25.573325 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.573862 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.575010 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.575427 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.575572 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:25.575630 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:25.575811 (+ 181us) server_negotiation.cc:299] Negotiation successful
0806 02:45:25.575895 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":66,"threads_started":1}
I20260806 02:45:25.576078 5180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:25.572265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:37335 (local address 127.1.183.130:34665)
0806 02:45:25.572615 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:25.572630 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:25.572744 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:25.573030 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:25.573033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:25.573059 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:25.573207 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:25.573212 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.573989 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.573996 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:25.574867 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:25.574876 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.575572 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.575576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:25.575671 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:25.575699 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 02:45:25.575792 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 02:45:25.575868 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":21,"spinlock_wait_cycles":896,"thread_start_us":65,"threads_started":1}
I20260806 02:45:25.576085 5179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:25.572190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:46263 (local address 127.1.183.130:54263)
0806 02:45:25.572546 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:25.572568 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:25.572715 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:25.572989 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:25.572992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:25.573007 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:25.573416 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:25.573422 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.574159 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.574164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:25.574937 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:25.574945 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.575598 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.575602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:25.575711 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:25.575729 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 02:45:25.575824 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 02:45:25.575903 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":20,"thread_start_us":102,"threads_started":1}
I20260806 02:45:25.576944 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "196e14e9ce3d4d4091be80aaa440e611" candidate_uuid: "6642632003e04747a05e7229c96f4582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff00848052f43e4b76c6b9c02e26f02" is_pre_election: true
I20260806 02:45:25.577204 5036 raft_consensus.cc:2454] T 196e14e9ce3d4d4091be80aaa440e611 P 6ff00848052f43e4b76c6b9c02e26f02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6642632003e04747a05e7229c96f4582 in term 0.
I20260806 02:45:25.577570 4600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "196e14e9ce3d4d4091be80aaa440e611" candidate_uuid: "6642632003e04747a05e7229c96f4582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5862084e3b914d4d877216c1c0ac63eb" is_pre_election: true
I20260806 02:45:25.577719 4600 raft_consensus.cc:2454] T 196e14e9ce3d4d4091be80aaa440e611 P 5862084e3b914d4d877216c1c0ac63eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6642632003e04747a05e7229c96f4582 in term 0.
I20260806 02:45:25.577704 4705 leader_election.cc:302] T 196e14e9ce3d4d4091be80aaa440e611 P 6642632003e04747a05e7229c96f4582 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6642632003e04747a05e7229c96f4582, 6ff00848052f43e4b76c6b9c02e26f02; no voters:
I20260806 02:45:25.578078 5155 raft_consensus.cc:2781] T 196e14e9ce3d4d4091be80aaa440e611 P 6642632003e04747a05e7229c96f4582 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:45:25.578146 5155 raft_consensus.cc:482] T 196e14e9ce3d4d4091be80aaa440e611 P 6642632003e04747a05e7229c96f4582 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:45:25.578174 5155 raft_consensus.cc:3037] T 196e14e9ce3d4d4091be80aaa440e611 P 6642632003e04747a05e7229c96f4582 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:25.579329 5155 raft_consensus.cc:504] T 196e14e9ce3d4d4091be80aaa440e611 P 6642632003e04747a05e7229c96f4582 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:25.579744 5155 leader_election.cc:288] T 196e14e9ce3d4d4091be80aaa440e611 P 6642632003e04747a05e7229c96f4582 [CANDIDATE]: Term 1 election: Requested vote from peers 5862084e3b914d4d877216c1c0ac63eb (127.1.183.129:46263), 6ff00848052f43e4b76c6b9c02e26f02 (127.1.183.131:37335)
I20260806 02:45:25.580130 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "196e14e9ce3d4d4091be80aaa440e611" candidate_uuid: "6642632003e04747a05e7229c96f4582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff00848052f43e4b76c6b9c02e26f02"
I20260806 02:45:25.580130 4600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "196e14e9ce3d4d4091be80aaa440e611" candidate_uuid: "6642632003e04747a05e7229c96f4582" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5862084e3b914d4d877216c1c0ac63eb"
I20260806 02:45:25.580291 5036 raft_consensus.cc:3037] T 196e14e9ce3d4d4091be80aaa440e611 P 6ff00848052f43e4b76c6b9c02e26f02 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:25.580291 4600 raft_consensus.cc:3037] T 196e14e9ce3d4d4091be80aaa440e611 P 5862084e3b914d4d877216c1c0ac63eb [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:25.581653 5036 raft_consensus.cc:2454] T 196e14e9ce3d4d4091be80aaa440e611 P 6ff00848052f43e4b76c6b9c02e26f02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6642632003e04747a05e7229c96f4582 in term 1.
I20260806 02:45:25.581972 4705 leader_election.cc:302] T 196e14e9ce3d4d4091be80aaa440e611 P 6642632003e04747a05e7229c96f4582 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6642632003e04747a05e7229c96f4582, 6ff00848052f43e4b76c6b9c02e26f02; no voters:
I20260806 02:45:25.582110 4600 raft_consensus.cc:2454] T 196e14e9ce3d4d4091be80aaa440e611 P 5862084e3b914d4d877216c1c0ac63eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6642632003e04747a05e7229c96f4582 in term 1.
I20260806 02:45:25.582173 5155 raft_consensus.cc:2781] T 196e14e9ce3d4d4091be80aaa440e611 P 6642632003e04747a05e7229c96f4582 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:25.582383 5155 raft_consensus.cc:686] T 196e14e9ce3d4d4091be80aaa440e611 P 6642632003e04747a05e7229c96f4582 [term 1 LEADER]: Becoming Leader. State: Replica: 6642632003e04747a05e7229c96f4582, State: Running, Role: LEADER
I20260806 02:45:25.582759 5155 consensus_queue.cc:237] T 196e14e9ce3d4d4091be80aaa440e611 P 6642632003e04747a05e7229c96f4582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:25.585543 5182 tablet_replica.cc:439] T 196e14e9ce3d4d4091be80aaa440e611 P 6642632003e04747a05e7229c96f4582: TxnStatusTablet state changed. Reason: New leader 6642632003e04747a05e7229c96f4582. Latest consensus state: current_term: 1 leader_uuid: "6642632003e04747a05e7229c96f4582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } } }
I20260806 02:45:25.585664 5182 tablet_replica.cc:442] T 196e14e9ce3d4d4091be80aaa440e611 P 6642632003e04747a05e7229c96f4582: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:45:25.586035 5184 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 02:45:25.586270 5184 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 02:45:25.586752 4361 catalog_manager.cc:5495] T 196e14e9ce3d4d4091be80aaa440e611 P 6642632003e04747a05e7229c96f4582 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6642632003e04747a05e7229c96f4582 (127.1.183.130). New cstate: current_term: 1 leader_uuid: "6642632003e04747a05e7229c96f4582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } health_report { overall_health: UNKNOWN } } }
I20260806 02:45:25.613564 5138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:25.609828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50568 (local address 127.1.183.130:34573)
0806 02:45:25.610005 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:25.610010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:25.610026 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:45:25.610144 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:25.610147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:25.610204 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:25.610301 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:25.610576 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.611096 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.611971 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.612257 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.612363 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:25.612686 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:25.613143 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:25.613274 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:25.613348 (+ 74us) server_negotiation.cc:299] Negotiation successful
0806 02:45:25.613407 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 02:45:25.613562 5171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:25.609718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:34573 (local address 127.0.0.1:50568)
0806 02:45:25.609861 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:25.609879 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:25.609994 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:25.610330 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:25.610333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:25.610341 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:45:25.610463 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:25.610468 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.611209 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.611214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:25.611840 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:25.611848 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.612377 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.612380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:25.612486 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:25.612532 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:45:25.613244 (+ 712us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:45:25.613262 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 02:45:25.613338 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 02:45:25.613443 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260806 02:45:25.619323 5036 raft_consensus.cc:1261] T 196e14e9ce3d4d4091be80aaa440e611 P 6ff00848052f43e4b76c6b9c02e26f02 [term 1 FOLLOWER]: Refusing update from remote peer 6642632003e04747a05e7229c96f4582: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:25.619969 4600 raft_consensus.cc:1261] T 196e14e9ce3d4d4091be80aaa440e611 P 5862084e3b914d4d877216c1c0ac63eb [term 1 FOLLOWER]: Refusing update from remote peer 6642632003e04747a05e7229c96f4582: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:25.619989 5155 consensus_queue.cc:1035] T 196e14e9ce3d4d4091be80aaa440e611 P 6642632003e04747a05e7229c96f4582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:25.620646 5155 consensus_queue.cc:1035] T 196e14e9ce3d4d4091be80aaa440e611 P 6642632003e04747a05e7229c96f4582 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:25.625137 5154 tablet_replica.cc:439] T 196e14e9ce3d4d4091be80aaa440e611 P 6ff00848052f43e4b76c6b9c02e26f02: TxnStatusTablet state changed. Reason: New leader 6642632003e04747a05e7229c96f4582. Latest consensus state: current_term: 1 leader_uuid: "6642632003e04747a05e7229c96f4582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } } }
I20260806 02:45:25.625275 5154 tablet_replica.cc:442] T 196e14e9ce3d4d4091be80aaa440e611 P 6ff00848052f43e4b76c6b9c02e26f02: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:25.625198 5157 tablet_replica.cc:439] T 196e14e9ce3d4d4091be80aaa440e611 P 5862084e3b914d4d877216c1c0ac63eb: TxnStatusTablet state changed. Reason: New leader 6642632003e04747a05e7229c96f4582. Latest consensus state: current_term: 1 leader_uuid: "6642632003e04747a05e7229c96f4582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } } }
I20260806 02:45:25.625330 5157 tablet_replica.cc:442] T 196e14e9ce3d4d4091be80aaa440e611 P 5862084e3b914d4d877216c1c0ac63eb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:25.627300 5187 mvcc.cc:205] Tried to move back new op lower bound from 7315391797729341440 to 7315391797590085632. Current Snapshot: MvccSnapshot[applied={T|T < 7315391797729341440}]
I20260806 02:45:25.627514 5155 tablet_replica.cc:439] T 196e14e9ce3d4d4091be80aaa440e611 P 6642632003e04747a05e7229c96f4582: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6642632003e04747a05e7229c96f4582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } } }
I20260806 02:45:25.627629 5155 tablet_replica.cc:442] T 196e14e9ce3d4d4091be80aaa440e611 P 6642632003e04747a05e7229c96f4582: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:45:25.627661 5154 tablet_replica.cc:439] T 196e14e9ce3d4d4091be80aaa440e611 P 6ff00848052f43e4b76c6b9c02e26f02: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6642632003e04747a05e7229c96f4582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } } }
I20260806 02:45:25.627760 5154 tablet_replica.cc:442] T 196e14e9ce3d4d4091be80aaa440e611 P 6ff00848052f43e4b76c6b9c02e26f02: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:25.629249 5188 mvcc.cc:205] Tried to move back new op lower bound from 7315391797729341440 to 7315391797590085632. Current Snapshot: MvccSnapshot[applied={T|T < 7315391797729341440}]
I20260806 02:45:25.633316 5182 tablet_replica.cc:439] T 196e14e9ce3d4d4091be80aaa440e611 P 6642632003e04747a05e7229c96f4582: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6642632003e04747a05e7229c96f4582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } } }
I20260806 02:45:25.633453 5182 tablet_replica.cc:442] T 196e14e9ce3d4d4091be80aaa440e611 P 6642632003e04747a05e7229c96f4582: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:45:25.635306 5157 tablet_replica.cc:439] T 196e14e9ce3d4d4091be80aaa440e611 P 5862084e3b914d4d877216c1c0ac63eb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6642632003e04747a05e7229c96f4582" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } } }
I20260806 02:45:25.635448 5157 tablet_replica.cc:442] T 196e14e9ce3d4d4091be80aaa440e611 P 5862084e3b914d4d877216c1c0ac63eb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:25.640480 5181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:25.635497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46360 (local address 127.1.183.131:37335)
0806 02:45:25.635662 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:25.635667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:25.636377 (+ 710us) server_negotiation.cc:407] Connection header received
0806 02:45:25.636642 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:25.636645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:25.636716 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:25.636814 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:25.637221 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.637844 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.638921 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.639157 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.639261 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:25.639552 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:25.640008 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:25.640136 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:25.640232 (+ 96us) server_negotiation.cc:299] Negotiation successful
0806 02:45:25.640297 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 02:45:25.652592 5138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:25.649006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:40159 (local address 127.1.183.130:34573)
0806 02:45:25.649155 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:25.649160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:25.649299 (+ 139us) server_negotiation.cc:407] Connection header received
0806 02:45:25.649465 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:25.649473 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:25.649521 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:25.649611 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:25.649888 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.650423 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.651364 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.651594 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.651711 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:25.652052 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:25.652126 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:25.652265 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:25.652385 (+ 120us) server_negotiation.cc:299] Negotiation successful
0806 02:45:25.652450 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 02:45:25.652612 5209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:25.648808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:34573 (local address 127.1.183.131:40159)
0806 02:45:25.649160 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:25.649177 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:25.649329 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:25.649624 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:25.649627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:25.649635 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:45:25.649778 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:25.649784 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.650538 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.650544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:25.651235 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:25.651244 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.651732 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.651736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:25.651832 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:25.651878 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:45:25.652265 (+ 387us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:45:25.652275 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 02:45:25.652346 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 02:45:25.652441 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":112,"threads_started":1}
I20260806 02:45:25.660491 4600 raft_consensus.cc:1261] T b1e292126bf64650af110300e1c420de P 5862084e3b914d4d877216c1c0ac63eb [term 1 FOLLOWER]: Refusing update from remote peer 6ff00848052f43e4b76c6b9c02e26f02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:25.660768 4808 raft_consensus.cc:1261] T b1e292126bf64650af110300e1c420de P 6642632003e04747a05e7229c96f4582 [term 1 FOLLOWER]: Refusing update from remote peer 6ff00848052f43e4b76c6b9c02e26f02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:25.661057 5154 consensus_queue.cc:1035] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:25.661413 5161 consensus_queue.cc:1035] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:25.666113 5189 mvcc.cc:205] Tried to move back new op lower bound from 7315391797900349440 to 7315391796758618112. Current Snapshot: MvccSnapshot[applied={T|T < 7315391797900349440}]
I20260806 02:45:25.690502 5181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:25.685833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:50497 (local address 127.1.183.131:37335)
0806 02:45:25.685985 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:25.685992 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:25.686240 (+ 248us) server_negotiation.cc:407] Connection header received
0806 02:45:25.686346 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:25.686349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:25.686424 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:25.686517 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:25.686785 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.687602 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.688844 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.689209 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.689337 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:25.689905 (+ 568us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:25.690000 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:25.690054 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:25.690229 (+ 175us) server_negotiation.cc:299] Negotiation successful
0806 02:45:25.690326 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:45:25.690992 5215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:25.685748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:37335 (local address 127.1.183.130:50497)
0806 02:45:25.686112 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:25.686145 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:25.686255 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:25.686536 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:25.686539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:25.686546 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:45:25.686661 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:25.686666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.687747 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.687750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:25.688720 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:25.688733 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.689330 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.689345 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:25.689445 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:25.689747 (+ 302us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:45:25.690128 (+ 381us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:45:25.690137 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 02:45:25.690741 (+ 604us) client_negotiation.cc:240] Negotiation successful
0806 02:45:25.690831 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":147,"threads_started":1}
I20260806 02:45:25.727367 4360 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:25.727626 4360 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:25.730913 4360 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 612964c3ba9140fea152adcd8e22048c: Sending DeleteTablet for 3 replicas of tablet b1e292126bf64650af110300e1c420de
I20260806 02:45:25.732539 4562 tablet_service.cc:1514] Processing DeleteTablet for tablet b1e292126bf64650af110300e1c420de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:25 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:45856
I20260806 02:45:25.733177 5219 tablet_replica.cc:330] T b1e292126bf64650af110300e1c420de P 5862084e3b914d4d877216c1c0ac63eb: stopping tablet replica
I20260806 02:45:25.733144 5004 tablet_service.cc:1514] Processing DeleteTablet for tablet b1e292126bf64650af110300e1c420de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:25 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:46356
I20260806 02:45:25.733765 5219 raft_consensus.cc:2229] T b1e292126bf64650af110300e1c420de P 5862084e3b914d4d877216c1c0ac63eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:25.733888 4781 tablet_service.cc:1514] Processing DeleteTablet for tablet b1e292126bf64650af110300e1c420de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:25 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:50558
I20260806 02:45:25.734136 5219 raft_consensus.cc:2258] T b1e292126bf64650af110300e1c420de P 5862084e3b914d4d877216c1c0ac63eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:25.734452 4360 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:45:25.734949 4360 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:45:25.735918 5219 ts_tablet_manager.cc:1905] T b1e292126bf64650af110300e1c420de P 5862084e3b914d4d877216c1c0ac63eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:25.739310 5219 ts_tablet_manager.cc:1918] T b1e292126bf64650af110300e1c420de P 5862084e3b914d4d877216c1c0ac63eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:45:25.739647 5221 tablet_replica.cc:330] T b1e292126bf64650af110300e1c420de P 6642632003e04747a05e7229c96f4582: stopping tablet replica
I20260806 02:45:25.739814 5219 log.cc:1196] T b1e292126bf64650af110300e1c420de P 5862084e3b914d4d877216c1c0ac63eb: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-0/wal/wals/b1e292126bf64650af110300e1c420de
I20260806 02:45:25.740255 5219 ts_tablet_manager.cc:1939] T b1e292126bf64650af110300e1c420de P 5862084e3b914d4d877216c1c0ac63eb: Deleting consensus metadata
I20260806 02:45:25.740661 5221 raft_consensus.cc:2229] T b1e292126bf64650af110300e1c420de P 6642632003e04747a05e7229c96f4582 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:25.741314 5004 tablet_service.cc:1467] Processing CreateTablet for tablet 69b95f2af9ff4daaa6e92c4379137201 (DEFAULT_TABLE table=test-table [id=2fe6ace9f3f64e3f93df9436856d5e8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:25.741643 5221 raft_consensus.cc:2258] T b1e292126bf64650af110300e1c420de P 6642632003e04747a05e7229c96f4582 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:25.741681 5004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69b95f2af9ff4daaa6e92c4379137201. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:25.742635 4342 catalog_manager.cc:4841] TS 5862084e3b914d4d877216c1c0ac63eb (127.1.183.129:46263): tablet b1e292126bf64650af110300e1c420de (table test-table [id=028f0b220bbc47c39080e1875dc0504b]) successfully deleted
I20260806 02:45:25.743881 5221 ts_tablet_manager.cc:1905] T b1e292126bf64650af110300e1c420de P 6642632003e04747a05e7229c96f4582: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:25.744525 4562 tablet_service.cc:1467] Processing CreateTablet for tablet 69b95f2af9ff4daaa6e92c4379137201 (DEFAULT_TABLE table=test-table [id=2fe6ace9f3f64e3f93df9436856d5e8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:25.744804 4562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69b95f2af9ff4daaa6e92c4379137201. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:25.747304 5221 ts_tablet_manager.cc:1918] T b1e292126bf64650af110300e1c420de P 6642632003e04747a05e7229c96f4582: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:45:25.747354 5144 tablet_bootstrap.cc:492] T 69b95f2af9ff4daaa6e92c4379137201 P 5862084e3b914d4d877216c1c0ac63eb: Bootstrap starting.
I20260806 02:45:25.747485 5221 log.cc:1196] T b1e292126bf64650af110300e1c420de P 6642632003e04747a05e7229c96f4582: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-1/wal/wals/b1e292126bf64650af110300e1c420de
I20260806 02:45:25.747948 5221 ts_tablet_manager.cc:1939] T b1e292126bf64650af110300e1c420de P 6642632003e04747a05e7229c96f4582: Deleting consensus metadata
I20260806 02:45:25.748098 4781 tablet_service.cc:1467] Processing CreateTablet for tablet 69b95f2af9ff4daaa6e92c4379137201 (DEFAULT_TABLE table=test-table [id=2fe6ace9f3f64e3f93df9436856d5e8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:25.748442 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69b95f2af9ff4daaa6e92c4379137201. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:25.748535 5144 tablet_bootstrap.cc:654] T 69b95f2af9ff4daaa6e92c4379137201 P 5862084e3b914d4d877216c1c0ac63eb: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:25.749589 4342 catalog_manager.cc:4841] TS 6642632003e04747a05e7229c96f4582 (127.1.183.130:34573): tablet b1e292126bf64650af110300e1c420de (table test-table [id=028f0b220bbc47c39080e1875dc0504b]) successfully deleted
I20260806 02:45:25.751009 5145 tablet_bootstrap.cc:492] T 69b95f2af9ff4daaa6e92c4379137201 P 6642632003e04747a05e7229c96f4582: Bootstrap starting.
I20260806 02:45:25.752352 5145 tablet_bootstrap.cc:654] T 69b95f2af9ff4daaa6e92c4379137201 P 6642632003e04747a05e7229c96f4582: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:25.754271 5148 tablet_bootstrap.cc:492] T 69b95f2af9ff4daaa6e92c4379137201 P 6ff00848052f43e4b76c6b9c02e26f02: Bootstrap starting.
I20260806 02:45:25.755702 5148 tablet_bootstrap.cc:654] T 69b95f2af9ff4daaa6e92c4379137201 P 6ff00848052f43e4b76c6b9c02e26f02: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:25.758345 5144 tablet_bootstrap.cc:492] T 69b95f2af9ff4daaa6e92c4379137201 P 5862084e3b914d4d877216c1c0ac63eb: No bootstrap required, opened a new log
I20260806 02:45:25.758508 5144 ts_tablet_manager.cc:1397] T 69b95f2af9ff4daaa6e92c4379137201 P 5862084e3b914d4d877216c1c0ac63eb: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260806 02:45:25.759166 5144 raft_consensus.cc:348] T 69b95f2af9ff4daaa6e92c4379137201 P 5862084e3b914d4d877216c1c0ac63eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:25.759327 5144 raft_consensus.cc:374] T 69b95f2af9ff4daaa6e92c4379137201 P 5862084e3b914d4d877216c1c0ac63eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:25.759366 5144 raft_consensus.cc:729] T 69b95f2af9ff4daaa6e92c4379137201 P 5862084e3b914d4d877216c1c0ac63eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5862084e3b914d4d877216c1c0ac63eb, State: Initialized, Role: FOLLOWER
I20260806 02:45:25.759521 5144 consensus_queue.cc:260] T 69b95f2af9ff4daaa6e92c4379137201 P 5862084e3b914d4d877216c1c0ac63eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:25.759886 5144 ts_tablet_manager.cc:1428] T 69b95f2af9ff4daaa6e92c4379137201 P 5862084e3b914d4d877216c1c0ac63eb: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 02:45:25.761013 5145 tablet_bootstrap.cc:492] T 69b95f2af9ff4daaa6e92c4379137201 P 6642632003e04747a05e7229c96f4582: No bootstrap required, opened a new log
I20260806 02:45:25.761106 5145 ts_tablet_manager.cc:1397] T 69b95f2af9ff4daaa6e92c4379137201 P 6642632003e04747a05e7229c96f4582: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260806 02:45:25.761277 5148 tablet_bootstrap.cc:492] T 69b95f2af9ff4daaa6e92c4379137201 P 6ff00848052f43e4b76c6b9c02e26f02: No bootstrap required, opened a new log
I20260806 02:45:25.761389 5148 ts_tablet_manager.cc:1397] T 69b95f2af9ff4daaa6e92c4379137201 P 6ff00848052f43e4b76c6b9c02e26f02: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20260806 02:45:25.761569 5145 raft_consensus.cc:348] T 69b95f2af9ff4daaa6e92c4379137201 P 6642632003e04747a05e7229c96f4582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:25.761689 5145 raft_consensus.cc:374] T 69b95f2af9ff4daaa6e92c4379137201 P 6642632003e04747a05e7229c96f4582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:25.761716 5145 raft_consensus.cc:729] T 69b95f2af9ff4daaa6e92c4379137201 P 6642632003e04747a05e7229c96f4582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6642632003e04747a05e7229c96f4582, State: Initialized, Role: FOLLOWER
I20260806 02:45:25.761829 5145 consensus_queue.cc:260] T 69b95f2af9ff4daaa6e92c4379137201 P 6642632003e04747a05e7229c96f4582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:25.761931 5148 raft_consensus.cc:348] T 69b95f2af9ff4daaa6e92c4379137201 P 6ff00848052f43e4b76c6b9c02e26f02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:25.762060 5145 ts_tablet_manager.cc:1428] T 69b95f2af9ff4daaa6e92c4379137201 P 6642632003e04747a05e7229c96f4582: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 02:45:25.762058 5148 raft_consensus.cc:374] T 69b95f2af9ff4daaa6e92c4379137201 P 6ff00848052f43e4b76c6b9c02e26f02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:25.762094 5148 raft_consensus.cc:729] T 69b95f2af9ff4daaa6e92c4379137201 P 6ff00848052f43e4b76c6b9c02e26f02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff00848052f43e4b76c6b9c02e26f02, State: Initialized, Role: FOLLOWER
I20260806 02:45:25.762219 5148 consensus_queue.cc:260] T 69b95f2af9ff4daaa6e92c4379137201 P 6ff00848052f43e4b76c6b9c02e26f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:25.762530 5148 ts_tablet_manager.cc:1428] T 69b95f2af9ff4daaa6e92c4379137201 P 6ff00848052f43e4b76c6b9c02e26f02: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 02:45:25.775904 5220 tablet_replica.cc:330] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02: stopping tablet replica
I20260806 02:45:25.776363 5220 raft_consensus.cc:2229] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:45:25.776816 5220 raft_consensus.cc:2258] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:25.778339 5220 ts_tablet_manager.cc:1905] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:25.782249 5220 ts_tablet_manager.cc:1918] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:45:25.782373 5220 log.cc:1196] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-2/wal/wals/b1e292126bf64650af110300e1c420de
I20260806 02:45:25.782739 5220 ts_tablet_manager.cc:1939] T b1e292126bf64650af110300e1c420de P 6ff00848052f43e4b76c6b9c02e26f02: Deleting consensus metadata
I20260806 02:45:25.784050 4341 catalog_manager.cc:4841] TS 6ff00848052f43e4b76c6b9c02e26f02 (127.1.183.131:37335): tablet b1e292126bf64650af110300e1c420de (table test-table [id=028f0b220bbc47c39080e1875dc0504b]) successfully deleted
I20260806 02:45:25.960309 5157 raft_consensus.cc:482] T 69b95f2af9ff4daaa6e92c4379137201 P 5862084e3b914d4d877216c1c0ac63eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:45:25.960457 5157 raft_consensus.cc:504] T 69b95f2af9ff4daaa6e92c4379137201 P 5862084e3b914d4d877216c1c0ac63eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:25.961746 5157 leader_election.cc:288] T 69b95f2af9ff4daaa6e92c4379137201 P 5862084e3b914d4d877216c1c0ac63eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6642632003e04747a05e7229c96f4582 (127.1.183.130:34573), 6ff00848052f43e4b76c6b9c02e26f02 (127.1.183.131:37335)
I20260806 02:45:25.966884 5138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:25.962591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:37645 (local address 127.1.183.130:34573)
0806 02:45:25.962731 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:25.962736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:25.962950 (+ 214us) server_negotiation.cc:407] Connection header received
0806 02:45:25.963153 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:25.963160 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:25.963232 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:25.963309 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:25.963925 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.964529 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.965633 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.966120 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.966246 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:25.966317 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:25.966636 (+ 319us) server_negotiation.cc:299] Negotiation successful
0806 02:45:25.966734 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 02:45:25.967059 5230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:25.962046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:34573 (local address 127.1.183.129:37645)
0806 02:45:25.962817 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:25.962837 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:25.962981 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:25.963496 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:25.963501 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:25.963525 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:25.963718 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:25.963736 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.964651 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.964656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:25.965485 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:25.965495 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.966280 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.966286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:25.966453 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:25.966500 (+ 47us) client_negotiation.cc:769] Sending connection context
0806 02:45:25.966598 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 02:45:25.966709 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":147,"threads_started":1}
I20260806 02:45:25.967479 5181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:25.962139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:33375 (local address 127.1.183.131:37335)
0806 02:45:25.962296 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:25.962302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:25.962636 (+ 334us) server_negotiation.cc:407] Connection header received
0806 02:45:25.962892 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:25.962894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:25.962972 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:25.963073 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:25.963414 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.964009 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.966114 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.966654 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.966807 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:25.966853 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:25.967250 (+ 397us) server_negotiation.cc:299] Negotiation successful
0806 02:45:25.967348 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:45:25.967700 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69b95f2af9ff4daaa6e92c4379137201" candidate_uuid: "5862084e3b914d4d877216c1c0ac63eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6642632003e04747a05e7229c96f4582" is_pre_election: true
I20260806 02:45:25.967875 5229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:25.962034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:37335 (local address 127.1.183.129:33375)
0806 02:45:25.962519 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:25.962535 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:25.962731 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:25.963088 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:25.963091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:25.963110 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:25.963274 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:25.963291 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.964142 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.964147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:25.965949 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:25.965962 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:25.966965 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:25.966972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:25.967094 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:25.967125 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 02:45:25.967627 (+ 502us) client_negotiation.cc:240] Negotiation successful
0806 02:45:25.967711 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":158,"threads_started":1}
I20260806 02:45:25.967862 4808 raft_consensus.cc:2454] T 69b95f2af9ff4daaa6e92c4379137201 P 6642632003e04747a05e7229c96f4582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5862084e3b914d4d877216c1c0ac63eb in term 0.
I20260806 02:45:25.968415 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69b95f2af9ff4daaa6e92c4379137201" candidate_uuid: "5862084e3b914d4d877216c1c0ac63eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff00848052f43e4b76c6b9c02e26f02" is_pre_election: true
I20260806 02:45:25.968561 5036 raft_consensus.cc:2454] T 69b95f2af9ff4daaa6e92c4379137201 P 6ff00848052f43e4b76c6b9c02e26f02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5862084e3b914d4d877216c1c0ac63eb in term 0.
I20260806 02:45:25.968564 4495 leader_election.cc:302] T 69b95f2af9ff4daaa6e92c4379137201 P 5862084e3b914d4d877216c1c0ac63eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5862084e3b914d4d877216c1c0ac63eb, 6642632003e04747a05e7229c96f4582; no voters:
I20260806 02:45:25.968868 5157 raft_consensus.cc:2781] T 69b95f2af9ff4daaa6e92c4379137201 P 5862084e3b914d4d877216c1c0ac63eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:45:25.968925 5157 raft_consensus.cc:482] T 69b95f2af9ff4daaa6e92c4379137201 P 5862084e3b914d4d877216c1c0ac63eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:45:25.968962 5157 raft_consensus.cc:3037] T 69b95f2af9ff4daaa6e92c4379137201 P 5862084e3b914d4d877216c1c0ac63eb [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:25.970474 5157 raft_consensus.cc:504] T 69b95f2af9ff4daaa6e92c4379137201 P 5862084e3b914d4d877216c1c0ac63eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:25.970827 5157 leader_election.cc:288] T 69b95f2af9ff4daaa6e92c4379137201 P 5862084e3b914d4d877216c1c0ac63eb [CANDIDATE]: Term 1 election: Requested vote from peers 6642632003e04747a05e7229c96f4582 (127.1.183.130:34573), 6ff00848052f43e4b76c6b9c02e26f02 (127.1.183.131:37335)
I20260806 02:45:25.971226 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69b95f2af9ff4daaa6e92c4379137201" candidate_uuid: "5862084e3b914d4d877216c1c0ac63eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6642632003e04747a05e7229c96f4582"
I20260806 02:45:25.971215 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69b95f2af9ff4daaa6e92c4379137201" candidate_uuid: "5862084e3b914d4d877216c1c0ac63eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff00848052f43e4b76c6b9c02e26f02"
I20260806 02:45:25.971364 5036 raft_consensus.cc:3037] T 69b95f2af9ff4daaa6e92c4379137201 P 6ff00848052f43e4b76c6b9c02e26f02 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:25.971355 4808 raft_consensus.cc:3037] T 69b95f2af9ff4daaa6e92c4379137201 P 6642632003e04747a05e7229c96f4582 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:25.972590 5036 raft_consensus.cc:2454] T 69b95f2af9ff4daaa6e92c4379137201 P 6ff00848052f43e4b76c6b9c02e26f02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5862084e3b914d4d877216c1c0ac63eb in term 1.
I20260806 02:45:25.972590 4808 raft_consensus.cc:2454] T 69b95f2af9ff4daaa6e92c4379137201 P 6642632003e04747a05e7229c96f4582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5862084e3b914d4d877216c1c0ac63eb in term 1.
I20260806 02:45:25.972910 4495 leader_election.cc:302] T 69b95f2af9ff4daaa6e92c4379137201 P 5862084e3b914d4d877216c1c0ac63eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5862084e3b914d4d877216c1c0ac63eb, 6642632003e04747a05e7229c96f4582; no voters:
I20260806 02:45:25.973119 5157 raft_consensus.cc:2781] T 69b95f2af9ff4daaa6e92c4379137201 P 5862084e3b914d4d877216c1c0ac63eb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:25.973373 5157 raft_consensus.cc:686] T 69b95f2af9ff4daaa6e92c4379137201 P 5862084e3b914d4d877216c1c0ac63eb [term 1 LEADER]: Becoming Leader. State: Replica: 5862084e3b914d4d877216c1c0ac63eb, State: Running, Role: LEADER
I20260806 02:45:25.973645 5157 consensus_queue.cc:237] T 69b95f2af9ff4daaa6e92c4379137201 P 5862084e3b914d4d877216c1c0ac63eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:25.976549 4362 catalog_manager.cc:5495] T 69b95f2af9ff4daaa6e92c4379137201 P 5862084e3b914d4d877216c1c0ac63eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 5862084e3b914d4d877216c1c0ac63eb (127.1.183.129). New cstate: current_term: 1 leader_uuid: "5862084e3b914d4d877216c1c0ac63eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } health_report { overall_health: UNKNOWN } } }
I20260806 02:45:26.021073 5143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:26.017093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45862 (local address 127.1.183.129:46263)
0806 02:45:26.017220 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:26.017227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:26.017243 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:45:26.017473 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:26.017476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:26.017539 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:26.017661 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:26.017951 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:26.018563 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:26.019409 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:26.019657 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:26.019774 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:26.020082 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:26.020646 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:26.020782 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:26.020854 (+ 72us) server_negotiation.cc:299] Negotiation successful
0806 02:45:26.020922 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:45:26.034032 5138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:26.030029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:41205 (local address 127.1.183.130:34573)
0806 02:45:26.030194 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:26.030202 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:26.030300 (+ 98us) server_negotiation.cc:407] Connection header received
0806 02:45:26.030427 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:26.030433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:26.030507 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:26.030608 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:26.030845 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:26.031793 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:26.032826 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:26.033077 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:26.033198 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:26.033519 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:26.033608 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:26.033691 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:26.033809 (+ 118us) server_negotiation.cc:299] Negotiation successful
0806 02:45:26.033870 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 02:45:26.034060 5236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:26.029889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:34573 (local address 127.1.183.129:41205)
0806 02:45:26.030209 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:26.030222 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:26.030322 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:26.030621 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:26.030623 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:26.030630 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:45:26.030739 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:26.030743 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:26.031971 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:26.031974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:26.032684 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:26.032695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:26.033205 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:26.033209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:26.033308 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:26.033360 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:45:26.033697 (+ 337us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:45:26.033707 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 02:45:26.033795 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 02:45:26.033888 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":115,"threads_started":1}
I20260806 02:45:26.041224 4808 raft_consensus.cc:1261] T 69b95f2af9ff4daaa6e92c4379137201 P 6642632003e04747a05e7229c96f4582 [term 1 FOLLOWER]: Refusing update from remote peer 5862084e3b914d4d877216c1c0ac63eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:26.041260 5036 raft_consensus.cc:1261] T 69b95f2af9ff4daaa6e92c4379137201 P 6ff00848052f43e4b76c6b9c02e26f02 [term 1 FOLLOWER]: Refusing update from remote peer 5862084e3b914d4d877216c1c0ac63eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:26.041797 5232 consensus_queue.cc:1035] T 69b95f2af9ff4daaa6e92c4379137201 P 5862084e3b914d4d877216c1c0ac63eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:26.042012 5157 consensus_queue.cc:1035] T 69b95f2af9ff4daaa6e92c4379137201 P 5862084e3b914d4d877216c1c0ac63eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:26.063043 5143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:26.059621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:56361 (local address 127.1.183.129:46263)
0806 02:45:26.059762 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:26.059769 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:26.059785 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:45:26.059836 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:26.059842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:26.059909 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:26.059997 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:26.060308 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:26.060991 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:26.061981 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:26.062228 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:26.062333 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:26.062594 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:26.062674 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:26.062767 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:26.062841 (+ 74us) server_negotiation.cc:299] Negotiation successful
0806 02:45:26.062893 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 02:45:26.063043 5215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:26.059376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:46263 (local address 127.1.183.130:56361)
0806 02:45:26.059534 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:26.059551 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:26.059638 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:26.060010 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:26.060013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:26.060022 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:45:26.060132 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:26.060138 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:26.061119 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:26.061122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:26.061852 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:26.061861 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:26.062343 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:26.062346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:26.062433 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:26.062461 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:45:26.062757 (+ 296us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:45:26.062765 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 02:45:26.062830 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 02:45:26.062916 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 02:45:26.064116 5189 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 02:45:27.633875 4362 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:27.634119 4362 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:27.636891 4362 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 612964c3ba9140fea152adcd8e22048c: Sending DeleteTablet for 3 replicas of tablet 69b95f2af9ff4daaa6e92c4379137201
I20260806 02:45:27.637544 4562 tablet_service.cc:1514] Processing DeleteTablet for tablet 69b95f2af9ff4daaa6e92c4379137201 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:27 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:45856
I20260806 02:45:27.637630 4780 tablet_service.cc:1514] Processing DeleteTablet for tablet 69b95f2af9ff4daaa6e92c4379137201 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:27 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:50558
I20260806 02:45:27.637730 5004 tablet_service.cc:1514] Processing DeleteTablet for tablet 69b95f2af9ff4daaa6e92c4379137201 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:27 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:46356
I20260806 02:45:27.637986 5270 tablet_replica.cc:330] T 69b95f2af9ff4daaa6e92c4379137201 P 5862084e3b914d4d877216c1c0ac63eb: stopping tablet replica
I20260806 02:45:27.638087 5272 tablet_replica.cc:330] T 69b95f2af9ff4daaa6e92c4379137201 P 6ff00848052f43e4b76c6b9c02e26f02: stopping tablet replica
I20260806 02:45:27.638155 5270 raft_consensus.cc:2229] T 69b95f2af9ff4daaa6e92c4379137201 P 5862084e3b914d4d877216c1c0ac63eb [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:45:27.638648 5270 raft_consensus.cc:2258] T 69b95f2af9ff4daaa6e92c4379137201 P 5862084e3b914d4d877216c1c0ac63eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:27.639351 5270 ts_tablet_manager.cc:1905] T 69b95f2af9ff4daaa6e92c4379137201 P 5862084e3b914d4d877216c1c0ac63eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:27.640636 5271 tablet_replica.cc:330] T 69b95f2af9ff4daaa6e92c4379137201 P 6642632003e04747a05e7229c96f4582: stopping tablet replica
I20260806 02:45:27.640869 5271 raft_consensus.cc:2229] T 69b95f2af9ff4daaa6e92c4379137201 P 6642632003e04747a05e7229c96f4582 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:27.641075 5271 raft_consensus.cc:2258] T 69b95f2af9ff4daaa6e92c4379137201 P 6642632003e04747a05e7229c96f4582 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:27.641856 5271 ts_tablet_manager.cc:1905] T 69b95f2af9ff4daaa6e92c4379137201 P 6642632003e04747a05e7229c96f4582: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:27.642686 5270 ts_tablet_manager.cc:1918] T 69b95f2af9ff4daaa6e92c4379137201 P 5862084e3b914d4d877216c1c0ac63eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:45:27.642822 5270 log.cc:1196] T 69b95f2af9ff4daaa6e92c4379137201 P 5862084e3b914d4d877216c1c0ac63eb: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-0/wal/wals/69b95f2af9ff4daaa6e92c4379137201
I20260806 02:45:27.638270 5272 raft_consensus.cc:2229] T 69b95f2af9ff4daaa6e92c4379137201 P 6ff00848052f43e4b76c6b9c02e26f02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:27.643165 5270 ts_tablet_manager.cc:1939] T 69b95f2af9ff4daaa6e92c4379137201 P 5862084e3b914d4d877216c1c0ac63eb: Deleting consensus metadata
I20260806 02:45:27.643219 5272 raft_consensus.cc:2258] T 69b95f2af9ff4daaa6e92c4379137201 P 6ff00848052f43e4b76c6b9c02e26f02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:27.643993 5272 ts_tablet_manager.cc:1905] T 69b95f2af9ff4daaa6e92c4379137201 P 6ff00848052f43e4b76c6b9c02e26f02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:27.644114 4342 catalog_manager.cc:4841] TS 5862084e3b914d4d877216c1c0ac63eb (127.1.183.129:46263): tablet 69b95f2af9ff4daaa6e92c4379137201 (table test-table [id=2fe6ace9f3f64e3f93df9436856d5e8a]) successfully deleted
I20260806 02:45:27.645143 5271 ts_tablet_manager.cc:1918] T 69b95f2af9ff4daaa6e92c4379137201 P 6642632003e04747a05e7229c96f4582: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:45:27.645232 5271 log.cc:1196] T 69b95f2af9ff4daaa6e92c4379137201 P 6642632003e04747a05e7229c96f4582: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-1/wal/wals/69b95f2af9ff4daaa6e92c4379137201
I20260806 02:45:27.645623 5271 ts_tablet_manager.cc:1939] T 69b95f2af9ff4daaa6e92c4379137201 P 6642632003e04747a05e7229c96f4582: Deleting consensus metadata
I20260806 02:45:27.646497 4342 catalog_manager.cc:4841] TS 6642632003e04747a05e7229c96f4582 (127.1.183.130:34573): tablet 69b95f2af9ff4daaa6e92c4379137201 (table test-table [id=2fe6ace9f3f64e3f93df9436856d5e8a]) successfully deleted
I20260806 02:45:27.647009 5272 ts_tablet_manager.cc:1918] T 69b95f2af9ff4daaa6e92c4379137201 P 6ff00848052f43e4b76c6b9c02e26f02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:45:27.647109 5272 log.cc:1196] T 69b95f2af9ff4daaa6e92c4379137201 P 6ff00848052f43e4b76c6b9c02e26f02: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-2/wal/wals/69b95f2af9ff4daaa6e92c4379137201
I20260806 02:45:27.647361 5272 ts_tablet_manager.cc:1939] T 69b95f2af9ff4daaa6e92c4379137201 P 6ff00848052f43e4b76c6b9c02e26f02: Deleting consensus metadata
I20260806 02:45:27.648131 4341 catalog_manager.cc:4841] TS 6ff00848052f43e4b76c6b9c02e26f02 (127.1.183.131:37335): tablet 69b95f2af9ff4daaa6e92c4379137201 (table test-table [id=2fe6ace9f3f64e3f93df9436856d5e8a]) successfully deleted
Aug 06 02:45:27 dist-test-slave-gmjp krb5kdc[4294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984327, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:45:27 dist-test-slave-gmjp krb5kdc[4294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984327, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:27.668845 5281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:27.659634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59602 (local address 127.1.183.190:41751)
0806 02:45:27.659971 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:27.659978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:27.659993 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:45:27.660119 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:27.660121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:27.660224 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:27.660310 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:27.661025 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:27.661592 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:27.662467 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:27.662690 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:27.662834 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:27.665429 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:27.665432 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:27.665435 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:27.665460 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:27.667041 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:27.667509 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:27.667512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:27.667513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:27.667599 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:27.667870 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:27.667872 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:27.667874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:27.668066 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:27.668269 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:27.668518 (+ 249us) server_negotiation.cc:299] Negotiation successful
0806 02:45:27.668656 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":152,"threads_started":1}
I20260806 02:45:27.671802 4362 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59602:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:45:27.672330 4362 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:45:27.677356 4562 tablet_service.cc:1467] Processing CreateTablet for tablet 7b168f5c87d443c2bd6ac3d6f899772a (DEFAULT_TABLE table=test-table [id=2724dfd59f4746ab8a718615eb437040]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:27.677438 4780 tablet_service.cc:1467] Processing CreateTablet for tablet 7b168f5c87d443c2bd6ac3d6f899772a (DEFAULT_TABLE table=test-table [id=2724dfd59f4746ab8a718615eb437040]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:27.677433 5004 tablet_service.cc:1467] Processing CreateTablet for tablet 7b168f5c87d443c2bd6ac3d6f899772a (DEFAULT_TABLE table=test-table [id=2724dfd59f4746ab8a718615eb437040]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:27.677727 5004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b168f5c87d443c2bd6ac3d6f899772a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:27.677728 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b168f5c87d443c2bd6ac3d6f899772a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:27.677731 4562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b168f5c87d443c2bd6ac3d6f899772a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:27.680377 5282 tablet_bootstrap.cc:492] T 7b168f5c87d443c2bd6ac3d6f899772a P 5862084e3b914d4d877216c1c0ac63eb: Bootstrap starting.
I20260806 02:45:27.680613 5287 tablet_bootstrap.cc:492] T 7b168f5c87d443c2bd6ac3d6f899772a P 6642632003e04747a05e7229c96f4582: Bootstrap starting.
I20260806 02:45:27.681316 5282 tablet_bootstrap.cc:654] T 7b168f5c87d443c2bd6ac3d6f899772a P 5862084e3b914d4d877216c1c0ac63eb: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:27.681393 5287 tablet_bootstrap.cc:654] T 7b168f5c87d443c2bd6ac3d6f899772a P 6642632003e04747a05e7229c96f4582: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:27.681980 5284 tablet_bootstrap.cc:492] T 7b168f5c87d443c2bd6ac3d6f899772a P 6ff00848052f43e4b76c6b9c02e26f02: Bootstrap starting.
I20260806 02:45:27.682614 5287 tablet_bootstrap.cc:492] T 7b168f5c87d443c2bd6ac3d6f899772a P 6642632003e04747a05e7229c96f4582: No bootstrap required, opened a new log
I20260806 02:45:27.682731 5287 ts_tablet_manager.cc:1397] T 7b168f5c87d443c2bd6ac3d6f899772a P 6642632003e04747a05e7229c96f4582: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 02:45:27.682845 5284 tablet_bootstrap.cc:654] T 7b168f5c87d443c2bd6ac3d6f899772a P 6ff00848052f43e4b76c6b9c02e26f02: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:27.682928 5282 tablet_bootstrap.cc:492] T 7b168f5c87d443c2bd6ac3d6f899772a P 5862084e3b914d4d877216c1c0ac63eb: No bootstrap required, opened a new log
I20260806 02:45:27.683050 5282 ts_tablet_manager.cc:1397] T 7b168f5c87d443c2bd6ac3d6f899772a P 5862084e3b914d4d877216c1c0ac63eb: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260806 02:45:27.683233 5287 raft_consensus.cc:348] T 7b168f5c87d443c2bd6ac3d6f899772a P 6642632003e04747a05e7229c96f4582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:27.683389 5287 raft_consensus.cc:374] T 7b168f5c87d443c2bd6ac3d6f899772a P 6642632003e04747a05e7229c96f4582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:27.683447 5287 raft_consensus.cc:729] T 7b168f5c87d443c2bd6ac3d6f899772a P 6642632003e04747a05e7229c96f4582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6642632003e04747a05e7229c96f4582, State: Initialized, Role: FOLLOWER
I20260806 02:45:27.683527 5282 raft_consensus.cc:348] T 7b168f5c87d443c2bd6ac3d6f899772a P 5862084e3b914d4d877216c1c0ac63eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:27.683633 5287 consensus_queue.cc:260] T 7b168f5c87d443c2bd6ac3d6f899772a P 6642632003e04747a05e7229c96f4582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:27.683699 5282 raft_consensus.cc:374] T 7b168f5c87d443c2bd6ac3d6f899772a P 5862084e3b914d4d877216c1c0ac63eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:27.683748 5282 raft_consensus.cc:729] T 7b168f5c87d443c2bd6ac3d6f899772a P 5862084e3b914d4d877216c1c0ac63eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5862084e3b914d4d877216c1c0ac63eb, State: Initialized, Role: FOLLOWER
I20260806 02:45:27.683889 5282 consensus_queue.cc:260] T 7b168f5c87d443c2bd6ac3d6f899772a P 5862084e3b914d4d877216c1c0ac63eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:27.684119 5287 ts_tablet_manager.cc:1428] T 7b168f5c87d443c2bd6ac3d6f899772a P 6642632003e04747a05e7229c96f4582: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 02:45:27.684305 5282 ts_tablet_manager.cc:1428] T 7b168f5c87d443c2bd6ac3d6f899772a P 5862084e3b914d4d877216c1c0ac63eb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 02:45:27.684460 5284 tablet_bootstrap.cc:492] T 7b168f5c87d443c2bd6ac3d6f899772a P 6ff00848052f43e4b76c6b9c02e26f02: No bootstrap required, opened a new log
I20260806 02:45:27.684571 5284 ts_tablet_manager.cc:1397] T 7b168f5c87d443c2bd6ac3d6f899772a P 6ff00848052f43e4b76c6b9c02e26f02: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260806 02:45:27.685042 5284 raft_consensus.cc:348] T 7b168f5c87d443c2bd6ac3d6f899772a P 6ff00848052f43e4b76c6b9c02e26f02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:27.685178 5284 raft_consensus.cc:374] T 7b168f5c87d443c2bd6ac3d6f899772a P 6ff00848052f43e4b76c6b9c02e26f02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:27.685213 5284 raft_consensus.cc:729] T 7b168f5c87d443c2bd6ac3d6f899772a P 6ff00848052f43e4b76c6b9c02e26f02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff00848052f43e4b76c6b9c02e26f02, State: Initialized, Role: FOLLOWER
I20260806 02:45:27.685431 5284 consensus_queue.cc:260] T 7b168f5c87d443c2bd6ac3d6f899772a P 6ff00848052f43e4b76c6b9c02e26f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:27.685863 5284 ts_tablet_manager.cc:1428] T 7b168f5c87d443c2bd6ac3d6f899772a P 6ff00848052f43e4b76c6b9c02e26f02: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 02:45:27.695225 5232 raft_consensus.cc:482] T 7b168f5c87d443c2bd6ac3d6f899772a P 5862084e3b914d4d877216c1c0ac63eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:45:27.695361 5232 raft_consensus.cc:504] T 7b168f5c87d443c2bd6ac3d6f899772a P 5862084e3b914d4d877216c1c0ac63eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:27.695930 5232 leader_election.cc:288] T 7b168f5c87d443c2bd6ac3d6f899772a P 5862084e3b914d4d877216c1c0ac63eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6642632003e04747a05e7229c96f4582 (127.1.183.130:34573), 6ff00848052f43e4b76c6b9c02e26f02 (127.1.183.131:37335)
I20260806 02:45:27.696355 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b168f5c87d443c2bd6ac3d6f899772a" candidate_uuid: "5862084e3b914d4d877216c1c0ac63eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6642632003e04747a05e7229c96f4582" is_pre_election: true
I20260806 02:45:27.696414 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b168f5c87d443c2bd6ac3d6f899772a" candidate_uuid: "5862084e3b914d4d877216c1c0ac63eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff00848052f43e4b76c6b9c02e26f02" is_pre_election: true
I20260806 02:45:27.696545 4808 raft_consensus.cc:2454] T 7b168f5c87d443c2bd6ac3d6f899772a P 6642632003e04747a05e7229c96f4582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5862084e3b914d4d877216c1c0ac63eb in term 0.
I20260806 02:45:27.696604 5036 raft_consensus.cc:2454] T 7b168f5c87d443c2bd6ac3d6f899772a P 6ff00848052f43e4b76c6b9c02e26f02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5862084e3b914d4d877216c1c0ac63eb in term 0.
I20260806 02:45:27.696833 4495 leader_election.cc:302] T 7b168f5c87d443c2bd6ac3d6f899772a P 5862084e3b914d4d877216c1c0ac63eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5862084e3b914d4d877216c1c0ac63eb, 6642632003e04747a05e7229c96f4582; no voters:
I20260806 02:45:27.697083 5232 raft_consensus.cc:2781] T 7b168f5c87d443c2bd6ac3d6f899772a P 5862084e3b914d4d877216c1c0ac63eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:45:27.697132 5232 raft_consensus.cc:482] T 7b168f5c87d443c2bd6ac3d6f899772a P 5862084e3b914d4d877216c1c0ac63eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:45:27.697158 5232 raft_consensus.cc:3037] T 7b168f5c87d443c2bd6ac3d6f899772a P 5862084e3b914d4d877216c1c0ac63eb [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:27.698216 5232 raft_consensus.cc:504] T 7b168f5c87d443c2bd6ac3d6f899772a P 5862084e3b914d4d877216c1c0ac63eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:27.698592 5232 leader_election.cc:288] T 7b168f5c87d443c2bd6ac3d6f899772a P 5862084e3b914d4d877216c1c0ac63eb [CANDIDATE]: Term 1 election: Requested vote from peers 6642632003e04747a05e7229c96f4582 (127.1.183.130:34573), 6ff00848052f43e4b76c6b9c02e26f02 (127.1.183.131:37335)
I20260806 02:45:27.698946 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b168f5c87d443c2bd6ac3d6f899772a" candidate_uuid: "5862084e3b914d4d877216c1c0ac63eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6642632003e04747a05e7229c96f4582"
I20260806 02:45:27.699033 5036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b168f5c87d443c2bd6ac3d6f899772a" candidate_uuid: "5862084e3b914d4d877216c1c0ac63eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff00848052f43e4b76c6b9c02e26f02"
I20260806 02:45:27.699074 4808 raft_consensus.cc:3037] T 7b168f5c87d443c2bd6ac3d6f899772a P 6642632003e04747a05e7229c96f4582 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:27.699158 5036 raft_consensus.cc:3037] T 7b168f5c87d443c2bd6ac3d6f899772a P 6ff00848052f43e4b76c6b9c02e26f02 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:27.700310 4808 raft_consensus.cc:2454] T 7b168f5c87d443c2bd6ac3d6f899772a P 6642632003e04747a05e7229c96f4582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5862084e3b914d4d877216c1c0ac63eb in term 1.
I20260806 02:45:27.700311 5036 raft_consensus.cc:2454] T 7b168f5c87d443c2bd6ac3d6f899772a P 6ff00848052f43e4b76c6b9c02e26f02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5862084e3b914d4d877216c1c0ac63eb in term 1.
I20260806 02:45:27.700606 4495 leader_election.cc:302] T 7b168f5c87d443c2bd6ac3d6f899772a P 5862084e3b914d4d877216c1c0ac63eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5862084e3b914d4d877216c1c0ac63eb, 6642632003e04747a05e7229c96f4582; no voters:
I20260806 02:45:27.700758 5232 raft_consensus.cc:2781] T 7b168f5c87d443c2bd6ac3d6f899772a P 5862084e3b914d4d877216c1c0ac63eb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:27.700850 5232 raft_consensus.cc:686] T 7b168f5c87d443c2bd6ac3d6f899772a P 5862084e3b914d4d877216c1c0ac63eb [term 1 LEADER]: Becoming Leader. State: Replica: 5862084e3b914d4d877216c1c0ac63eb, State: Running, Role: LEADER
I20260806 02:45:27.700995 5232 consensus_queue.cc:237] T 7b168f5c87d443c2bd6ac3d6f899772a P 5862084e3b914d4d877216c1c0ac63eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:27.702853 4361 catalog_manager.cc:5495] T 7b168f5c87d443c2bd6ac3d6f899772a P 5862084e3b914d4d877216c1c0ac63eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 5862084e3b914d4d877216c1c0ac63eb (127.1.183.129). New cstate: current_term: 1 leader_uuid: "5862084e3b914d4d877216c1c0ac63eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } health_report { overall_health: UNKNOWN } } }
I20260806 02:45:27.721652 5297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:27.718386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45866 (local address 127.1.183.129:46263)
0806 02:45:27.718689 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:27.718696 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:27.718709 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:45:27.718751 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:27.718753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:27.718796 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:27.718882 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:27.719112 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:27.719638 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:27.720610 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:27.720867 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:27.720954 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:27.721248 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:27.721322 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:27.721376 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:27.721478 (+ 102us) server_negotiation.cc:299] Negotiation successful
0806 02:45:27.721526 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":144,"threads_started":1}
I20260806 02:45:27.728298 5036 raft_consensus.cc:1261] T 7b168f5c87d443c2bd6ac3d6f899772a P 6ff00848052f43e4b76c6b9c02e26f02 [term 1 FOLLOWER]: Refusing update from remote peer 5862084e3b914d4d877216c1c0ac63eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:27.728376 4808 raft_consensus.cc:1261] T 7b168f5c87d443c2bd6ac3d6f899772a P 6642632003e04747a05e7229c96f4582 [term 1 FOLLOWER]: Refusing update from remote peer 5862084e3b914d4d877216c1c0ac63eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:27.728731 5232 consensus_queue.cc:1035] T 7b168f5c87d443c2bd6ac3d6f899772a P 5862084e3b914d4d877216c1c0ac63eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:27.728905 5261 consensus_queue.cc:1035] T 7b168f5c87d443c2bd6ac3d6f899772a P 5862084e3b914d4d877216c1c0ac63eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:27.764331 4362 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:27.764492 4362 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:27.766613 4362 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 612964c3ba9140fea152adcd8e22048c: Sending DeleteTablet for 3 replicas of tablet 7b168f5c87d443c2bd6ac3d6f899772a
I20260806 02:45:27.767333 5004 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b168f5c87d443c2bd6ac3d6f899772a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:27 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:46356
I20260806 02:45:27.767359 4562 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b168f5c87d443c2bd6ac3d6f899772a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:27 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:45856
I20260806 02:45:27.767473 4780 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b168f5c87d443c2bd6ac3d6f899772a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:27 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:50558
I20260806 02:45:27.767555 5270 tablet_replica.cc:330] T 7b168f5c87d443c2bd6ac3d6f899772a P 5862084e3b914d4d877216c1c0ac63eb: stopping tablet replica
I20260806 02:45:27.767575 5272 tablet_replica.cc:330] T 7b168f5c87d443c2bd6ac3d6f899772a P 6ff00848052f43e4b76c6b9c02e26f02: stopping tablet replica
I20260806 02:45:27.767649 5271 tablet_replica.cc:330] T 7b168f5c87d443c2bd6ac3d6f899772a P 6642632003e04747a05e7229c96f4582: stopping tablet replica
I20260806 02:45:27.767674 5270 raft_consensus.cc:2229] T 7b168f5c87d443c2bd6ac3d6f899772a P 5862084e3b914d4d877216c1c0ac63eb [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:45:27.767701 5272 raft_consensus.cc:2229] T 7b168f5c87d443c2bd6ac3d6f899772a P 6ff00848052f43e4b76c6b9c02e26f02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:27.767758 5271 raft_consensus.cc:2229] T 7b168f5c87d443c2bd6ac3d6f899772a P 6642632003e04747a05e7229c96f4582 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:27.767819 5272 raft_consensus.cc:2258] T 7b168f5c87d443c2bd6ac3d6f899772a P 6ff00848052f43e4b76c6b9c02e26f02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:27.767861 5271 raft_consensus.cc:2258] T 7b168f5c87d443c2bd6ac3d6f899772a P 6642632003e04747a05e7229c96f4582 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:27.767892 5270 raft_consensus.cc:2258] T 7b168f5c87d443c2bd6ac3d6f899772a P 5862084e3b914d4d877216c1c0ac63eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:27.768646 5270 ts_tablet_manager.cc:1905] T 7b168f5c87d443c2bd6ac3d6f899772a P 5862084e3b914d4d877216c1c0ac63eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:27.768581 4362 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59602:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 02:45:27.768831 5271 ts_tablet_manager.cc:1905] T 7b168f5c87d443c2bd6ac3d6f899772a P 6642632003e04747a05e7229c96f4582: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 02:45:27.769014 4362 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:45:27.771400 5270 ts_tablet_manager.cc:1918] T 7b168f5c87d443c2bd6ac3d6f899772a P 5862084e3b914d4d877216c1c0ac63eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:45:27.771484 5270 log.cc:1196] T 7b168f5c87d443c2bd6ac3d6f899772a P 5862084e3b914d4d877216c1c0ac63eb: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-0/wal/wals/7b168f5c87d443c2bd6ac3d6f899772a
I20260806 02:45:27.771559 5271 ts_tablet_manager.cc:1918] T 7b168f5c87d443c2bd6ac3d6f899772a P 6642632003e04747a05e7229c96f4582: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:45:27.771632 5271 log.cc:1196] T 7b168f5c87d443c2bd6ac3d6f899772a P 6642632003e04747a05e7229c96f4582: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-1/wal/wals/7b168f5c87d443c2bd6ac3d6f899772a
I20260806 02:45:27.771760 5270 ts_tablet_manager.cc:1939] T 7b168f5c87d443c2bd6ac3d6f899772a P 5862084e3b914d4d877216c1c0ac63eb: Deleting consensus metadata
I20260806 02:45:27.771855 5271 ts_tablet_manager.cc:1939] T 7b168f5c87d443c2bd6ac3d6f899772a P 6642632003e04747a05e7229c96f4582: Deleting consensus metadata
I20260806 02:45:27.772482 4342 catalog_manager.cc:4841] TS 5862084e3b914d4d877216c1c0ac63eb (127.1.183.129:46263): tablet 7b168f5c87d443c2bd6ac3d6f899772a (table test-table [id=2724dfd59f4746ab8a718615eb437040]) successfully deleted
I20260806 02:45:27.772701 4342 catalog_manager.cc:4841] TS 6642632003e04747a05e7229c96f4582 (127.1.183.130:34573): tablet 7b168f5c87d443c2bd6ac3d6f899772a (table test-table [id=2724dfd59f4746ab8a718615eb437040]) successfully deleted
I20260806 02:45:27.773487 5272 ts_tablet_manager.cc:1905] T 7b168f5c87d443c2bd6ac3d6f899772a P 6ff00848052f43e4b76c6b9c02e26f02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:27.774286 4562 tablet_service.cc:1467] Processing CreateTablet for tablet 9d4c79cfa07d45c5adeef45500161900 (DEFAULT_TABLE table=test-table [id=5b8b1a536ba04cca8dced1c8cb430251]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:27.774569 4562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d4c79cfa07d45c5adeef45500161900. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:27.774576 5004 tablet_service.cc:1467] Processing CreateTablet for tablet 9d4c79cfa07d45c5adeef45500161900 (DEFAULT_TABLE table=test-table [id=5b8b1a536ba04cca8dced1c8cb430251]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:27.774627 4780 tablet_service.cc:1467] Processing CreateTablet for tablet 9d4c79cfa07d45c5adeef45500161900 (DEFAULT_TABLE table=test-table [id=5b8b1a536ba04cca8dced1c8cb430251]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:27.774845 5004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d4c79cfa07d45c5adeef45500161900. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:27.774856 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d4c79cfa07d45c5adeef45500161900. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:27.776057 5272 ts_tablet_manager.cc:1918] T 7b168f5c87d443c2bd6ac3d6f899772a P 6ff00848052f43e4b76c6b9c02e26f02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:45:27.776127 5272 log.cc:1196] T 7b168f5c87d443c2bd6ac3d6f899772a P 6ff00848052f43e4b76c6b9c02e26f02: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-2/wal/wals/7b168f5c87d443c2bd6ac3d6f899772a
I20260806 02:45:27.776456 5272 ts_tablet_manager.cc:1939] T 7b168f5c87d443c2bd6ac3d6f899772a P 6ff00848052f43e4b76c6b9c02e26f02: Deleting consensus metadata
I20260806 02:45:27.777019 5282 tablet_bootstrap.cc:492] T 9d4c79cfa07d45c5adeef45500161900 P 5862084e3b914d4d877216c1c0ac63eb: Bootstrap starting.
I20260806 02:45:27.777236 5287 tablet_bootstrap.cc:492] T 9d4c79cfa07d45c5adeef45500161900 P 6642632003e04747a05e7229c96f4582: Bootstrap starting.
I20260806 02:45:27.777434 4341 catalog_manager.cc:4841] TS 6ff00848052f43e4b76c6b9c02e26f02 (127.1.183.131:37335): tablet 7b168f5c87d443c2bd6ac3d6f899772a (table test-table [id=2724dfd59f4746ab8a718615eb437040]) successfully deleted
I20260806 02:45:27.777829 5284 tablet_bootstrap.cc:492] T 9d4c79cfa07d45c5adeef45500161900 P 6ff00848052f43e4b76c6b9c02e26f02: Bootstrap starting.
I20260806 02:45:27.777891 5282 tablet_bootstrap.cc:654] T 9d4c79cfa07d45c5adeef45500161900 P 5862084e3b914d4d877216c1c0ac63eb: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:27.778131 5287 tablet_bootstrap.cc:654] T 9d4c79cfa07d45c5adeef45500161900 P 6642632003e04747a05e7229c96f4582: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:27.778718 5284 tablet_bootstrap.cc:654] T 9d4c79cfa07d45c5adeef45500161900 P 6ff00848052f43e4b76c6b9c02e26f02: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:27.779351 5287 tablet_bootstrap.cc:492] T 9d4c79cfa07d45c5adeef45500161900 P 6642632003e04747a05e7229c96f4582: No bootstrap required, opened a new log
I20260806 02:45:27.779457 5287 ts_tablet_manager.cc:1397] T 9d4c79cfa07d45c5adeef45500161900 P 6642632003e04747a05e7229c96f4582: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:45:27.779816 5282 tablet_bootstrap.cc:492] T 9d4c79cfa07d45c5adeef45500161900 P 5862084e3b914d4d877216c1c0ac63eb: No bootstrap required, opened a new log
I20260806 02:45:27.779860 5284 tablet_bootstrap.cc:492] T 9d4c79cfa07d45c5adeef45500161900 P 6ff00848052f43e4b76c6b9c02e26f02: No bootstrap required, opened a new log
I20260806 02:45:27.779878 5282 ts_tablet_manager.cc:1397] T 9d4c79cfa07d45c5adeef45500161900 P 5862084e3b914d4d877216c1c0ac63eb: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260806 02:45:27.779947 5284 ts_tablet_manager.cc:1397] T 9d4c79cfa07d45c5adeef45500161900 P 6ff00848052f43e4b76c6b9c02e26f02: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260806 02:45:27.779908 5287 raft_consensus.cc:348] T 9d4c79cfa07d45c5adeef45500161900 P 6642632003e04747a05e7229c96f4582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:27.780021 5287 raft_consensus.cc:374] T 9d4c79cfa07d45c5adeef45500161900 P 6642632003e04747a05e7229c96f4582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:27.780055 5287 raft_consensus.cc:729] T 9d4c79cfa07d45c5adeef45500161900 P 6642632003e04747a05e7229c96f4582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6642632003e04747a05e7229c96f4582, State: Initialized, Role: FOLLOWER
I20260806 02:45:27.780218 5287 consensus_queue.cc:260] T 9d4c79cfa07d45c5adeef45500161900 P 6642632003e04747a05e7229c96f4582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:27.780292 5282 raft_consensus.cc:348] T 9d4c79cfa07d45c5adeef45500161900 P 5862084e3b914d4d877216c1c0ac63eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:27.780423 5282 raft_consensus.cc:374] T 9d4c79cfa07d45c5adeef45500161900 P 5862084e3b914d4d877216c1c0ac63eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:27.780457 5282 raft_consensus.cc:729] T 9d4c79cfa07d45c5adeef45500161900 P 5862084e3b914d4d877216c1c0ac63eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5862084e3b914d4d877216c1c0ac63eb, State: Initialized, Role: FOLLOWER
I20260806 02:45:27.780406 5284 raft_consensus.cc:348] T 9d4c79cfa07d45c5adeef45500161900 P 6ff00848052f43e4b76c6b9c02e26f02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:27.780511 5284 raft_consensus.cc:374] T 9d4c79cfa07d45c5adeef45500161900 P 6ff00848052f43e4b76c6b9c02e26f02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:27.780516 5287 ts_tablet_manager.cc:1428] T 9d4c79cfa07d45c5adeef45500161900 P 6642632003e04747a05e7229c96f4582: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:45:27.780550 5284 raft_consensus.cc:729] T 9d4c79cfa07d45c5adeef45500161900 P 6ff00848052f43e4b76c6b9c02e26f02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff00848052f43e4b76c6b9c02e26f02, State: Initialized, Role: FOLLOWER
I20260806 02:45:27.780578 5282 consensus_queue.cc:260] T 9d4c79cfa07d45c5adeef45500161900 P 5862084e3b914d4d877216c1c0ac63eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:27.780678 5284 consensus_queue.cc:260] T 9d4c79cfa07d45c5adeef45500161900 P 6ff00848052f43e4b76c6b9c02e26f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:27.780912 5282 ts_tablet_manager.cc:1428] T 9d4c79cfa07d45c5adeef45500161900 P 5862084e3b914d4d877216c1c0ac63eb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:45:27.780936 5284 ts_tablet_manager.cc:1428] T 9d4c79cfa07d45c5adeef45500161900 P 6ff00848052f43e4b76c6b9c02e26f02: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:45:27.838311 5292 raft_consensus.cc:482] T 9d4c79cfa07d45c5adeef45500161900 P 6ff00848052f43e4b76c6b9c02e26f02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:45:27.838515 5292 raft_consensus.cc:504] T 9d4c79cfa07d45c5adeef45500161900 P 6ff00848052f43e4b76c6b9c02e26f02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:27.838985 5292 leader_election.cc:288] T 9d4c79cfa07d45c5adeef45500161900 P 6ff00848052f43e4b76c6b9c02e26f02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5862084e3b914d4d877216c1c0ac63eb (127.1.183.129:46263), 6642632003e04747a05e7229c96f4582 (127.1.183.130:34573)
I20260806 02:45:27.839530 4600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d4c79cfa07d45c5adeef45500161900" candidate_uuid: "6ff00848052f43e4b76c6b9c02e26f02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5862084e3b914d4d877216c1c0ac63eb" is_pre_election: true
I20260806 02:45:27.839568 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d4c79cfa07d45c5adeef45500161900" candidate_uuid: "6ff00848052f43e4b76c6b9c02e26f02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6642632003e04747a05e7229c96f4582" is_pre_election: true
I20260806 02:45:27.839716 4600 raft_consensus.cc:2454] T 9d4c79cfa07d45c5adeef45500161900 P 5862084e3b914d4d877216c1c0ac63eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ff00848052f43e4b76c6b9c02e26f02 in term 0.
I20260806 02:45:27.839748 4808 raft_consensus.cc:2454] T 9d4c79cfa07d45c5adeef45500161900 P 6642632003e04747a05e7229c96f4582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ff00848052f43e4b76c6b9c02e26f02 in term 0.
I20260806 02:45:27.840122 4919 leader_election.cc:302] T 9d4c79cfa07d45c5adeef45500161900 P 6ff00848052f43e4b76c6b9c02e26f02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5862084e3b914d4d877216c1c0ac63eb, 6ff00848052f43e4b76c6b9c02e26f02; no voters:
I20260806 02:45:27.840348 5292 raft_consensus.cc:2781] T 9d4c79cfa07d45c5adeef45500161900 P 6ff00848052f43e4b76c6b9c02e26f02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:45:27.840430 5292 raft_consensus.cc:482] T 9d4c79cfa07d45c5adeef45500161900 P 6ff00848052f43e4b76c6b9c02e26f02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:45:27.840476 5292 raft_consensus.cc:3037] T 9d4c79cfa07d45c5adeef45500161900 P 6ff00848052f43e4b76c6b9c02e26f02 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:27.841543 5292 raft_consensus.cc:504] T 9d4c79cfa07d45c5adeef45500161900 P 6ff00848052f43e4b76c6b9c02e26f02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:27.841895 5292 leader_election.cc:288] T 9d4c79cfa07d45c5adeef45500161900 P 6ff00848052f43e4b76c6b9c02e26f02 [CANDIDATE]: Term 1 election: Requested vote from peers 5862084e3b914d4d877216c1c0ac63eb (127.1.183.129:46263), 6642632003e04747a05e7229c96f4582 (127.1.183.130:34573)
I20260806 02:45:27.842347 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d4c79cfa07d45c5adeef45500161900" candidate_uuid: "6ff00848052f43e4b76c6b9c02e26f02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6642632003e04747a05e7229c96f4582"
I20260806 02:45:27.842371 4600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d4c79cfa07d45c5adeef45500161900" candidate_uuid: "6ff00848052f43e4b76c6b9c02e26f02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5862084e3b914d4d877216c1c0ac63eb"
I20260806 02:45:27.842491 4808 raft_consensus.cc:3037] T 9d4c79cfa07d45c5adeef45500161900 P 6642632003e04747a05e7229c96f4582 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:27.842506 4600 raft_consensus.cc:3037] T 9d4c79cfa07d45c5adeef45500161900 P 5862084e3b914d4d877216c1c0ac63eb [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:27.843595 4808 raft_consensus.cc:2454] T 9d4c79cfa07d45c5adeef45500161900 P 6642632003e04747a05e7229c96f4582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ff00848052f43e4b76c6b9c02e26f02 in term 1.
I20260806 02:45:27.843605 4600 raft_consensus.cc:2454] T 9d4c79cfa07d45c5adeef45500161900 P 5862084e3b914d4d877216c1c0ac63eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ff00848052f43e4b76c6b9c02e26f02 in term 1.
I20260806 02:45:27.843915 4919 leader_election.cc:302] T 9d4c79cfa07d45c5adeef45500161900 P 6ff00848052f43e4b76c6b9c02e26f02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6642632003e04747a05e7229c96f4582, 6ff00848052f43e4b76c6b9c02e26f02; no voters:
I20260806 02:45:27.844100 5292 raft_consensus.cc:2781] T 9d4c79cfa07d45c5adeef45500161900 P 6ff00848052f43e4b76c6b9c02e26f02 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:27.844352 5292 raft_consensus.cc:686] T 9d4c79cfa07d45c5adeef45500161900 P 6ff00848052f43e4b76c6b9c02e26f02 [term 1 LEADER]: Becoming Leader. State: Replica: 6ff00848052f43e4b76c6b9c02e26f02, State: Running, Role: LEADER
I20260806 02:45:27.844524 5292 consensus_queue.cc:237] T 9d4c79cfa07d45c5adeef45500161900 P 6ff00848052f43e4b76c6b9c02e26f02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } }
I20260806 02:45:27.846199 4361 catalog_manager.cc:5495] T 9d4c79cfa07d45c5adeef45500161900 P 6ff00848052f43e4b76c6b9c02e26f02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ff00848052f43e4b76c6b9c02e26f02 (127.1.183.131). New cstate: current_term: 1 leader_uuid: "6ff00848052f43e4b76c6b9c02e26f02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ff00848052f43e4b76c6b9c02e26f02" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37335 } health_report { overall_health: HEALTHY } } }
I20260806 02:45:27.871449 5317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:27.868115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46364 (local address 127.1.183.131:37335)
0806 02:45:27.868493 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:27.868501 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:27.868513 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:45:27.868554 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:27.868556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:27.868598 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:27.868695 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:27.868970 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:27.869498 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:27.870313 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:27.870599 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:27.870707 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:27.870975 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:27.871058 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:27.871140 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:27.871229 (+ 89us) server_negotiation.cc:299] Negotiation successful
0806 02:45:27.871283 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":194,"threads_started":1}
I20260806 02:45:27.878108 4600 raft_consensus.cc:1261] T 9d4c79cfa07d45c5adeef45500161900 P 5862084e3b914d4d877216c1c0ac63eb [term 1 FOLLOWER]: Refusing update from remote peer 6ff00848052f43e4b76c6b9c02e26f02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:27.878087 4808 raft_consensus.cc:1261] T 9d4c79cfa07d45c5adeef45500161900 P 6642632003e04747a05e7229c96f4582 [term 1 FOLLOWER]: Refusing update from remote peer 6ff00848052f43e4b76c6b9c02e26f02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:27.878557 5312 consensus_queue.cc:1035] T 9d4c79cfa07d45c5adeef45500161900 P 6ff00848052f43e4b76c6b9c02e26f02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6642632003e04747a05e7229c96f4582" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:27.878733 5292 consensus_queue.cc:1035] T 9d4c79cfa07d45c5adeef45500161900 P 6ff00848052f43e4b76c6b9c02e26f02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5862084e3b914d4d877216c1c0ac63eb" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 02:45:27.893817 5247 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 02:45:29.448937 4361 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:29.449139 4361 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:29.451597 4361 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 612964c3ba9140fea152adcd8e22048c: Sending DeleteTablet for 3 replicas of tablet 9d4c79cfa07d45c5adeef45500161900
I20260806 02:45:29.452237 4562 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d4c79cfa07d45c5adeef45500161900 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:29 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:45856
I20260806 02:45:29.452394 5004 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d4c79cfa07d45c5adeef45500161900 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:29 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:46356
I20260806 02:45:29.452610 4780 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d4c79cfa07d45c5adeef45500161900 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:29 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:50558
I20260806 02:45:29.452757 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 4465
I20260806 02:45:29.452781 5359 tablet_replica.cc:330] T 9d4c79cfa07d45c5adeef45500161900 P 6ff00848052f43e4b76c6b9c02e26f02: stopping tablet replica
I20260806 02:45:29.456740 5360 tablet_replica.cc:330] T 9d4c79cfa07d45c5adeef45500161900 P 6642632003e04747a05e7229c96f4582: stopping tablet replica
I20260806 02:45:29.456907 5360 raft_consensus.cc:2229] T 9d4c79cfa07d45c5adeef45500161900 P 6642632003e04747a05e7229c96f4582 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:29.457038 5360 raft_consensus.cc:2258] T 9d4c79cfa07d45c5adeef45500161900 P 6642632003e04747a05e7229c96f4582 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:29.458297 5360 ts_tablet_manager.cc:1905] T 9d4c79cfa07d45c5adeef45500161900 P 6642632003e04747a05e7229c96f4582: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:29.461767 5360 ts_tablet_manager.cc:1918] T 9d4c79cfa07d45c5adeef45500161900 P 6642632003e04747a05e7229c96f4582: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:45:29.461917 5360 log.cc:1196] T 9d4c79cfa07d45c5adeef45500161900 P 6642632003e04747a05e7229c96f4582: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-1/wal/wals/9d4c79cfa07d45c5adeef45500161900
I20260806 02:45:29.462224 5360 ts_tablet_manager.cc:1939] T 9d4c79cfa07d45c5adeef45500161900 P 6642632003e04747a05e7229c96f4582: Deleting consensus metadata
I20260806 02:45:29.462488 5359 raft_consensus.cc:2229] T 9d4c79cfa07d45c5adeef45500161900 P 6ff00848052f43e4b76c6b9c02e26f02 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:45:29.462757 5359 raft_consensus.cc:2258] T 9d4c79cfa07d45c5adeef45500161900 P 6ff00848052f43e4b76c6b9c02e26f02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:29.463080 4342 catalog_manager.cc:4841] TS 6642632003e04747a05e7229c96f4582 (127.1.183.130:34573): tablet 9d4c79cfa07d45c5adeef45500161900 (table test-table [id=5b8b1a536ba04cca8dced1c8cb430251]) successfully deleted
I20260806 02:45:29.464084 5359 ts_tablet_manager.cc:1905] T 9d4c79cfa07d45c5adeef45500161900 P 6ff00848052f43e4b76c6b9c02e26f02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:29.466373 5359 ts_tablet_manager.cc:1918] T 9d4c79cfa07d45c5adeef45500161900 P 6ff00848052f43e4b76c6b9c02e26f02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:45:29.466449 5359 log.cc:1196] T 9d4c79cfa07d45c5adeef45500161900 P 6ff00848052f43e4b76c6b9c02e26f02: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984313887056-1758-0/minicluster-data/ts-2/wal/wals/9d4c79cfa07d45c5adeef45500161900
I20260806 02:45:29.466748 5359 ts_tablet_manager.cc:1939] T 9d4c79cfa07d45c5adeef45500161900 P 6ff00848052f43e4b76c6b9c02e26f02: Deleting consensus metadata
I20260806 02:45:29.467628 4341 catalog_manager.cc:4841] TS 6ff00848052f43e4b76c6b9c02e26f02 (127.1.183.131:37335): tablet 9d4c79cfa07d45c5adeef45500161900 (table test-table [id=5b8b1a536ba04cca8dced1c8cb430251]) successfully deleted
W20260806 02:45:29.471500 4342 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.183.129:46263 (error 108)
I20260806 02:45:29.471695 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 4683
I20260806 02:45:29.476238 5362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:29.474633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:46263 (local address 127.0.0.1:45876)
0806 02:45:29.475945 (+ 1312us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:29.476059 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.129:46263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1185,"thread_start_us":1029,"threads_started":1}
W20260806 02:45:29.480332 4342 catalog_manager.cc:4584] TS 5862084e3b914d4d877216c1c0ac63eb (127.1.183.129:46263): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9d4c79cfa07d45c5adeef45500161900: Network error: Client connection negotiation failed: client connection to 127.1.183.129:46263: connect: Connection refused (error 111)
I20260806 02:45:29.487202 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 4893
I20260806 02:45:29.507923 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 4312
2026-08-06T02:45:29Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6970 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:45:29 dist-test-slave-gmjp krb5kdc[5366](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:45:29 dist-test-slave-gmjp krb5kdc[5366](info): set up 2 sockets
Aug 06 02:45:29 dist-test-slave-gmjp krb5kdc[5366](info): commencing operation
krb5kdc: starting...
W20260806 02:45:30.750741 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.210s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:45:30 dist-test-slave-gmjp krb5kdc[5366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:45:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
I20260806 02:45:30.885515 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:44091
--webserver_interface=127.1.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:37755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:44091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:31.005650 5384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:31.006075 5384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:31.006165 5384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:31.010535 5384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:31.010668 5384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:31.010713 5384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:31.010761 5384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:31.010797 5384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:31.016605 5384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:37755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:44091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:44091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:31.017891 5384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:31.019053 5384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:31.028398 5396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:31.028484 5391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:31.028437 5394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:31.028545 5384 server_base.cc:1034] running on GCE node
I20260806 02:45:31.029374 5384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:31.030606 5384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:31.031826 5384 hybrid_clock.cc:648] HybridClock initialized: now 1785984331031788 us; error 50 us; skew 500 ppm
Aug 06 02:45:31 dist-test-slave-gmjp krb5kdc[5366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984331, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:31.034746 5384 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:45:31.035773 5384 webserver.cc:460] Webserver started at http://127.1.183.190:41479/ using document root <none> and password file <none>
I20260806 02:45:31.036418 5384 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:31.036489 5384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:31.036674 5384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:31.038843 5384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "98fca77a8bff449e805f0b575b5caf38"
format_stamp: "Formatted at 2026-08-06 02:45:31 on dist-test-slave-gmjp"
server_key: "77ff462a245110b24c9fba0b0f784e4c"
server_key_iv: "ba58475327ec804da23254599895e31f"
server_key_version: "encryptionkey@0"
I20260806 02:45:31.039412 5384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "98fca77a8bff449e805f0b575b5caf38"
format_stamp: "Formatted at 2026-08-06 02:45:31 on dist-test-slave-gmjp"
server_key: "77ff462a245110b24c9fba0b0f784e4c"
server_key_iv: "ba58475327ec804da23254599895e31f"
server_key_version: "encryptionkey@0"
I20260806 02:45:31.043469 5384 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:45:31.046087 5405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:31.047402 5384 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:45:31.047516 5384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "98fca77a8bff449e805f0b575b5caf38"
format_stamp: "Formatted at 2026-08-06 02:45:31 on dist-test-slave-gmjp"
server_key: "77ff462a245110b24c9fba0b0f784e4c"
server_key_iv: "ba58475327ec804da23254599895e31f"
server_key_version: "encryptionkey@0"
I20260806 02:45:31.047614 5384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:31.077704 5384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:31.081180 5384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:31.081354 5384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:31.089004 5384 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:44091
I20260806 02:45:31.089026 5484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:44091 every 8 connection(s)
I20260806 02:45:31.090090 5384 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:31.092957 5485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:31.093492 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 5384
I20260806 02:45:31.093618 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:45:31.093883 1758 external_mini_cluster.cc:1428] Setting key 5dd56c000e7b3a9866b5902125526466
I20260806 02:45:31.099771 5485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98fca77a8bff449e805f0b575b5caf38: Bootstrap starting.
I20260806 02:45:31.102276 5485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98fca77a8bff449e805f0b575b5caf38: Neither blocks nor log segments found. Creating new log.
Aug 06 02:45:31 dist-test-slave-gmjp krb5kdc[5366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:31.103134 5485 log.cc:824] T 00000000000000000000000000000000 P 98fca77a8bff449e805f0b575b5caf38: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:31.105758 5485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98fca77a8bff449e805f0b575b5caf38: No bootstrap required, opened a new log
I20260806 02:45:31.109056 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:31.095349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38850 (local address 127.1.183.190:44091)
0806 02:45:31.095824 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:31.095839 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:31.095876 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:45:31.096623 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:31.096644 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:31.096993 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:31.097343 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:31.098209 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:31.099220 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:31.100195 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:31.100678 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:31.100856 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:31.103475 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:31.103481 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:31.103495 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:31.103526 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:31.105576 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:31.106108 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:31.106117 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:31.106124 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:31.106282 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:31.106561 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:31.106564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:31.106565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:31.107105 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:31.107298 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:31.107526 (+ 228us) server_negotiation.cc:299] Negotiation successful
0806 02:45:31.107872 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":124,"threads_started":1}
I20260806 02:45:31.110623 5485 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98fca77a8bff449e805f0b575b5caf38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98fca77a8bff449e805f0b575b5caf38" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 44091 } }
I20260806 02:45:31.110906 5485 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98fca77a8bff449e805f0b575b5caf38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:31.110970 5485 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98fca77a8bff449e805f0b575b5caf38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98fca77a8bff449e805f0b575b5caf38, State: Initialized, Role: FOLLOWER
I20260806 02:45:31.111475 5485 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98fca77a8bff449e805f0b575b5caf38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98fca77a8bff449e805f0b575b5caf38" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 44091 } }
I20260806 02:45:31.111650 5485 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98fca77a8bff449e805f0b575b5caf38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:31.111724 5485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98fca77a8bff449e805f0b575b5caf38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:31.111828 5485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98fca77a8bff449e805f0b575b5caf38 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:31.112969 5485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98fca77a8bff449e805f0b575b5caf38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98fca77a8bff449e805f0b575b5caf38" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 44091 } }
I20260806 02:45:31.113344 5485 leader_election.cc:302] T 00000000000000000000000000000000 P 98fca77a8bff449e805f0b575b5caf38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98fca77a8bff449e805f0b575b5caf38; no voters:
I20260806 02:45:31.113705 5485 leader_election.cc:288] T 00000000000000000000000000000000 P 98fca77a8bff449e805f0b575b5caf38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:31.113868 5492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98fca77a8bff449e805f0b575b5caf38 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:31.114123 5492 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98fca77a8bff449e805f0b575b5caf38 [term 1 LEADER]: Becoming Leader. State: Replica: 98fca77a8bff449e805f0b575b5caf38, State: Running, Role: LEADER
I20260806 02:45:31.114511 5492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98fca77a8bff449e805f0b575b5caf38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98fca77a8bff449e805f0b575b5caf38" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 44091 } }
I20260806 02:45:31.115082 5485 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98fca77a8bff449e805f0b575b5caf38 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:31.117173 5494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98fca77a8bff449e805f0b575b5caf38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98fca77a8bff449e805f0b575b5caf38. Latest consensus state: current_term: 1 leader_uuid: "98fca77a8bff449e805f0b575b5caf38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98fca77a8bff449e805f0b575b5caf38" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 44091 } } }
I20260806 02:45:31.117334 5494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98fca77a8bff449e805f0b575b5caf38 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:31.117776 5493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98fca77a8bff449e805f0b575b5caf38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98fca77a8bff449e805f0b575b5caf38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98fca77a8bff449e805f0b575b5caf38" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 44091 } } }
I20260806 02:45:31.117875 5493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98fca77a8bff449e805f0b575b5caf38 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:31.118337 5511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:31.121584 5511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:31.128399 5511 catalog_manager.cc:1347] Generated new cluster ID: cfe15fca2e644038b6f8c5d111a0ed0e
I20260806 02:45:31.128505 5511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:31.142654 5511 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:31.143771 5511 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:31.154731 5511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98fca77a8bff449e805f0b575b5caf38: Generated new TSK 0
I20260806 02:45:31.155719 5511 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
I20260806 02:45:31.199285 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.129:0
--local_ip_for_outbound_sockets=127.1.183.129
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:44091
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:37755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:31.326131 5525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:31.326447 5525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:31.326494 5525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:31.330806 5525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:31.330889 5525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:31.330956 5525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.129
I20260806 02:45:31.336760 5525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:37755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:44091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.129
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:31.337960 5525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:31.339099 5525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:31.349768 5537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:31.349756 5539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:31.349763 5541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:31.350001 5525 server_base.cc:1034] running on GCE node
I20260806 02:45:31.350582 5525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:31.351380 5525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:31.352548 5525 hybrid_clock.cc:648] HybridClock initialized: now 1785984331352531 us; error 31 us; skew 500 ppm
Aug 06 02:45:31 dist-test-slave-gmjp krb5kdc[5366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984331, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:31.355594 5525 init.cc:377] Logged in from keytab as kudu/127.1.183.129@KRBTEST.COM (short username kudu)
I20260806 02:45:31.356662 5525 webserver.cc:460] Webserver started at http://127.1.183.129:36351/ using document root <none> and password file <none>
I20260806 02:45:31.357270 5525 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:31.357367 5525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:31.357612 5525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:31.359850 5525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/data/instance:
uuid: "acdc0d74035c4134a2af855b0dde6bda"
format_stamp: "Formatted at 2026-08-06 02:45:31 on dist-test-slave-gmjp"
server_key: "fc80fdbc9b957b3c1d403a70450d4ca2"
server_key_iv: "c6d8508d0ab0fd30f4089c7a548a32ac"
server_key_version: "encryptionkey@0"
I20260806 02:45:31.360489 5525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance:
uuid: "acdc0d74035c4134a2af855b0dde6bda"
format_stamp: "Formatted at 2026-08-06 02:45:31 on dist-test-slave-gmjp"
server_key: "fc80fdbc9b957b3c1d403a70450d4ca2"
server_key_iv: "c6d8508d0ab0fd30f4089c7a548a32ac"
server_key_version: "encryptionkey@0"
I20260806 02:45:31.364639 5525 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:45:31.367312 5554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:31.368793 5525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:31.368942 5525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/wal
uuid: "acdc0d74035c4134a2af855b0dde6bda"
format_stamp: "Formatted at 2026-08-06 02:45:31 on dist-test-slave-gmjp"
server_key: "fc80fdbc9b957b3c1d403a70450d4ca2"
server_key_iv: "c6d8508d0ab0fd30f4089c7a548a32ac"
server_key_version: "encryptionkey@0"
I20260806 02:45:31.369064 5525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:31.393049 5525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:31.397456 5525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:31.397701 5525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:31.398415 5525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:31.399600 5525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:31.399672 5525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:31.399747 5525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:31.399778 5525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:31.409695 5525 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.129:39837
I20260806 02:45:31.409715 5724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.129:39837 every 8 connection(s)
I20260806 02:45:31.410697 5525 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
I20260806 02:45:31.419026 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 5525
I20260806 02:45:31.419179 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance
I20260806 02:45:31.419456 1758 external_mini_cluster.cc:1428] Setting key d6aad796b1bf5116376a105a6f276688
Aug 06 02:45:31 dist-test-slave-gmjp krb5kdc[5366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984331, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:31.426890 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:31.412728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:40889 (local address 127.1.183.190:44091)
0806 02:45:31.412929 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:31.412939 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:31.413868 (+ 929us) server_negotiation.cc:407] Connection header received
0806 02:45:31.415432 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:31.415435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:31.415497 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:31.415612 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:31.417097 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:31.417708 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:31.418968 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:31.419290 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:31.419414 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:31.422524 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:31.422531 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:31.422537 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:31.422577 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:31.424324 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:31.425046 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:31.425049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:31.425050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:31.425126 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:31.425563 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:31.425565 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:31.425566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:31.425749 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:31.425826 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:31.426575 (+ 749us) server_negotiation.cc:299] Negotiation successful
0806 02:45:31.426737 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260806 02:45:31.428372 5730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:31.413135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:44091 (local address 127.1.183.129:40889)
0806 02:45:31.413653 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:31.413701 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:31.415183 (+ 1482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:31.415806 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:31.415814 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:31.416358 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:31.416936 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:31.416950 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:31.417857 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:31.417860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:31.418821 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:31.418832 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:31.419467 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:31.419471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:31.419683 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:31.420428 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:31.420446 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:31.422319 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:31.424455 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:31.424465 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:31.424491 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:31.424937 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:31.425226 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:31.425232 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:31.425238 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:31.425453 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:31.425867 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:31.425877 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:31.426270 (+ 393us) client_negotiation.cc:769] Sending connection context
0806 02:45:31.426542 (+ 272us) client_negotiation.cc:240] Negotiation successful
0806 02:45:31.426989 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":127,"threads_started":1}
I20260806 02:45:31.429780 5726 heartbeater.cc:347] Connected to a master server at 127.1.183.190:44091
I20260806 02:45:31.430083 5726 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:31.430752 5726 heartbeater.cc:511] Master 127.1.183.190:44091 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.183.130@KRBTEST.COM; defaulting to no policy
I20260806 02:45:31.432663 5433 ts_manager.cc:194] Registered new tserver with Master: acdc0d74035c4134a2af855b0dde6bda (127.1.183.129:39837)
I20260806 02:45:31.433995 5433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.129@KRBTEST.COM'} at 127.1.183.129:40889
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
I20260806 02:45:31.451167 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.130:0
--local_ip_for_outbound_sockets=127.1.183.130
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:44091
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:37755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:31.570191 5737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:31.570482 5737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:31.570524 5737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:31.574687 5737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:31.574753 5737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:31.574812 5737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.130
I20260806 02:45:31.580246 5737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:37755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:44091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.130
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:31.581339 5737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:31.582468 5737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:31.592998 5749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:31.593183 5737 server_base.cc:1034] running on GCE node
W20260806 02:45:31.593043 5743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:31.593036 5747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:31.593724 5737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:31.594429 5737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:31.595624 5737 hybrid_clock.cc:648] HybridClock initialized: now 1785984331595605 us; error 31 us; skew 500 ppm
Aug 06 02:45:31 dist-test-slave-gmjp krb5kdc[5366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984331, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:31.598551 5737 init.cc:377] Logged in from keytab as kudu/127.1.183.130@KRBTEST.COM (short username kudu)
I20260806 02:45:31.599435 5737 webserver.cc:460] Webserver started at http://127.1.183.130:43411/ using document root <none> and password file <none>
I20260806 02:45:31.600000 5737 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:31.600061 5737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:31.600310 5737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:31.602492 5737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/data/instance:
uuid: "575e01e18d1c4958b59bb066ced3e555"
format_stamp: "Formatted at 2026-08-06 02:45:31 on dist-test-slave-gmjp"
server_key: "274ff00e992e4cdd10220c0fb2faa51f"
server_key_iv: "5e60df18e6554d1194a65db453c0281f"
server_key_version: "encryptionkey@0"
I20260806 02:45:31.603065 5737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance:
uuid: "575e01e18d1c4958b59bb066ced3e555"
format_stamp: "Formatted at 2026-08-06 02:45:31 on dist-test-slave-gmjp"
server_key: "274ff00e992e4cdd10220c0fb2faa51f"
server_key_iv: "5e60df18e6554d1194a65db453c0281f"
server_key_version: "encryptionkey@0"
I20260806 02:45:31.607194 5737 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 02:45:31.609922 5761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:31.611213 5737 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:45:31.611349 5737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/wal
uuid: "575e01e18d1c4958b59bb066ced3e555"
format_stamp: "Formatted at 2026-08-06 02:45:31 on dist-test-slave-gmjp"
server_key: "274ff00e992e4cdd10220c0fb2faa51f"
server_key_iv: "5e60df18e6554d1194a65db453c0281f"
server_key_version: "encryptionkey@0"
I20260806 02:45:31.611467 5737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:31.637887 5737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:31.641845 5737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:31.642083 5737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:31.642758 5737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:31.643673 5737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:31.643723 5737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:31.643793 5737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:31.643831 5737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:31.653921 5737 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.130:43973
I20260806 02:45:31.653934 5944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.130:43973 every 8 connection(s)
I20260806 02:45:31.655033 5737 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
I20260806 02:45:31.659718 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 5737
I20260806 02:45:31.659842 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance
I20260806 02:45:31.660181 1758 external_mini_cluster.cc:1428] Setting key 0d65da24b30466f73a08262598d08f35
Aug 06 02:45:31 dist-test-slave-gmjp krb5kdc[5366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984331, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.183.131@KRBTEST.COM; defaulting to no policy
I20260806 02:45:31.670511 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:31.656966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:59899 (local address 127.1.183.190:44091)
0806 02:45:31.657122 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:31.657127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:31.658059 (+ 932us) server_negotiation.cc:407] Connection header received
0806 02:45:31.659383 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:31.659385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:31.659435 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:31.659495 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:31.661132 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:31.661843 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:31.663594 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:31.663851 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:31.663981 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:31.666663 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:31.666667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:31.666669 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:31.666703 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:31.668259 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:31.668841 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:31.668844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:31.668846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:31.668938 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:31.669308 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:31.669313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:31.669315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:31.669522 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:31.669630 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:31.670196 (+ 566us) server_negotiation.cc:299] Negotiation successful
0806 02:45:31.670338 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:45:31.671445 5950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:31.657450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:44091 (local address 127.1.183.130:59899)
0806 02:45:31.657893 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:31.657945 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:31.659184 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:31.659705 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:31.659714 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:31.660291 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:31.660981 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:31.661000 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:31.661976 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:31.661980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:31.663467 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:31.663482 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:31.664030 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:31.664034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:31.664270 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:31.664929 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:31.664949 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:31.666485 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:31.668384 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:31.668391 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:31.668410 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:31.668711 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:31.669038 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:31.669041 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:31.669043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:31.669206 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:31.669646 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:31.669655 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:31.669943 (+ 288us) client_negotiation.cc:769] Sending connection context
0806 02:45:31.670152 (+ 209us) client_negotiation.cc:240] Negotiation successful
0806 02:45:31.670538 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":104,"threads_started":1}
I20260806 02:45:31.672919 5945 heartbeater.cc:347] Connected to a master server at 127.1.183.190:44091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.131@KRBTEST.COM" created.
I20260806 02:45:31.673259 5945 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:31.673949 5945 heartbeater.cc:511] Master 127.1.183.190:44091 requested a full tablet report, sending...
I20260806 02:45:31.675211 5433 ts_manager.cc:194] Registered new tserver with Master: 575e01e18d1c4958b59bb066ced3e555 (127.1.183.130:43973)
I20260806 02:45:31.675915 5433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.130@KRBTEST.COM'} at 127.1.183.130:59899
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
I20260806 02:45:31.691148 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.131:0
--local_ip_for_outbound_sockets=127.1.183.131
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:44091
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:37755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:31.810681 5953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:31.811058 5953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:31.811139 5953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:31.815490 5953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:31.815594 5953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:31.815686 5953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.131
I20260806 02:45:31.821585 5953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:37755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:44091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.131
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:31.822829 5953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:31.823926 5953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:31.833930 5966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:31.833950 5964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:31.834234 5962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:31.834244 5953 server_base.cc:1034] running on GCE node
I20260806 02:45:31.834757 5953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:31.835446 5953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:31.836634 5953 hybrid_clock.cc:648] HybridClock initialized: now 1785984331836618 us; error 29 us; skew 500 ppm
Aug 06 02:45:31 dist-test-slave-gmjp krb5kdc[5366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984331, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:31.839562 5953 init.cc:377] Logged in from keytab as kudu/127.1.183.131@KRBTEST.COM (short username kudu)
I20260806 02:45:31.840564 5953 webserver.cc:460] Webserver started at http://127.1.183.131:46039/ using document root <none> and password file <none>
I20260806 02:45:31.841171 5953 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:31.841236 5953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:31.841486 5953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:31.843590 5953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/data/instance:
uuid: "b8d8e9766ade493b8a0b24e67fc0abe5"
format_stamp: "Formatted at 2026-08-06 02:45:31 on dist-test-slave-gmjp"
server_key: "4e4f9ee7274092e3d6a95a630c43cff6"
server_key_iv: "21cba167458e690c29d2de01409ba818"
server_key_version: "encryptionkey@0"
I20260806 02:45:31.844220 5953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance:
uuid: "b8d8e9766ade493b8a0b24e67fc0abe5"
format_stamp: "Formatted at 2026-08-06 02:45:31 on dist-test-slave-gmjp"
server_key: "4e4f9ee7274092e3d6a95a630c43cff6"
server_key_iv: "21cba167458e690c29d2de01409ba818"
server_key_version: "encryptionkey@0"
I20260806 02:45:31.848120 5953 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260806 02:45:31.850663 5978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:31.852035 5953 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:45:31.852228 5953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/wal
uuid: "b8d8e9766ade493b8a0b24e67fc0abe5"
format_stamp: "Formatted at 2026-08-06 02:45:31 on dist-test-slave-gmjp"
server_key: "4e4f9ee7274092e3d6a95a630c43cff6"
server_key_iv: "21cba167458e690c29d2de01409ba818"
server_key_version: "encryptionkey@0"
I20260806 02:45:31.852355 5953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:31.894073 5953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:31.897452 5953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:31.897684 5953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:31.898340 5953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:31.899252 5953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:31.899325 5953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:31.899392 5953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:31.899436 5953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:31.908833 5953 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.131:34509
I20260806 02:45:31.908852 6167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.131:34509 every 8 connection(s)
I20260806 02:45:31.909801 5953 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
Aug 06 02:45:31 dist-test-slave-gmjp krb5kdc[5366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984331, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:31.920049 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 5953
I20260806 02:45:31.920202 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance
I20260806 02:45:31.920584 1758 external_mini_cluster.cc:1428] Setting key 6465b4cd0d6ab8c9fc8370492669e5dc
I20260806 02:45:31.924407 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:31.911757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:41675 (local address 127.1.183.190:44091)
0806 02:45:31.911942 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:31.911948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:31.912888 (+ 940us) server_negotiation.cc:407] Connection header received
0806 02:45:31.914196 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:31.914199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:31.914250 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:31.914328 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:31.915608 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:31.916315 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:31.917524 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:31.917737 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:31.917819 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:31.920473 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:31.920478 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:31.920481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:31.920516 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:31.922308 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:31.922856 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:31.922860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:31.922861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:31.922973 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:31.923340 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:31.923342 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:31.923343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:31.923538 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:31.923618 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:31.924114 (+ 496us) server_negotiation.cc:299] Negotiation successful
0806 02:45:31.924281 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 02:45:31.925325 6174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:31.912247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:44091 (local address 127.1.183.131:41675)
0806 02:45:31.912723 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:31.912768 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:31.913992 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:31.914501 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:31.914507 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:31.914869 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:31.915464 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:31.915480 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:31.916440 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:31.916442 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:31.917398 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:31.917408 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:31.917873 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:31.917876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:31.918032 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:31.918603 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:31.918620 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:31.920219 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:31.922425 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:31.922430 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:31.922445 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:31.922728 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:31.923080 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:31.923083 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:31.923085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:31.923253 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:31.923631 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:31.923636 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:31.923889 (+ 253us) client_negotiation.cc:769] Sending connection context
0806 02:45:31.924061 (+ 172us) client_negotiation.cc:240] Negotiation successful
0806 02:45:31.924375 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":124,"threads_started":1}
I20260806 02:45:31.926465 6170 heartbeater.cc:347] Connected to a master server at 127.1.183.190:44091
I20260806 02:45:31.926726 6170 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:31.927279 6170 heartbeater.cc:511] Master 127.1.183.190:44091 requested a full tablet report, sending...
I20260806 02:45:31.928411 5433 ts_manager.cc:194] Registered new tserver with Master: b8d8e9766ade493b8a0b24e67fc0abe5 (127.1.183.131:34509)
I20260806 02:45:31.929052 5433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.131@KRBTEST.COM'} at 127.1.183.131:41675
I20260806 02:45:31.934864 1758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 02:45:31.950193 5489 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:31.948667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38860 (local address 127.1.183.190:44091)
0806 02:45:31.948871 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:31.948875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:31.948955 (+ 80us) server_negotiation.cc:407] Connection header received
0806 02:45:31.949081 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:31.949083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:31.949129 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:31.949212 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:31.950018 (+ 806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38860: BlockingRecv error: recv got EOF from 127.0.0.1:38860 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260806 02:45:31.951205 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 5525
I20260806 02:45:31.978514 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 5737
I20260806 02:45:31.997673 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 5953
I20260806 02:45:32.005944 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 5384
2026-08-06T02:45:32Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2494 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:45:32 dist-test-slave-gmjp krb5kdc[6189](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:45:32 dist-test-slave-gmjp krb5kdc[6189](info): set up 2 sockets
Aug 06 02:45:32 dist-test-slave-gmjp krb5kdc[6189](info): commencing operation
krb5kdc: starting...
W20260806 02:45:33.266775 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.231s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:45:33 dist-test-slave-gmjp krb5kdc[6189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:45:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
I20260806 02:45:33.400858 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:46457
--webserver_interface=127.1.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:46653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:46457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:33.522032 6213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:33.522423 6213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:33.522505 6213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:33.526757 6213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:33.526881 6213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:33.526919 6213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:33.526955 6213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:33.526996 6213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:33.532951 6213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:46653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:46457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:46457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:33.534199 6213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:33.535357 6213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:33.544302 6219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:33.544360 6213 server_base.cc:1034] running on GCE node
W20260806 02:45:33.544371 6221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:33.544605 6224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:33.545220 6213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:33.546397 6213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:33.547598 6213 hybrid_clock.cc:648] HybridClock initialized: now 1785984333547583 us; error 27 us; skew 500 ppm
Aug 06 02:45:33 dist-test-slave-gmjp krb5kdc[6189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984333, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:33.550689 6213 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:45:33.551780 6213 webserver.cc:460] Webserver started at http://127.1.183.190:41131/ using document root <none> and password file <none>
I20260806 02:45:33.552466 6213 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:33.552585 6213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:33.552834 6213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:33.555022 6213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "a8c6610947fb49239893ee32b0d313a7"
format_stamp: "Formatted at 2026-08-06 02:45:33 on dist-test-slave-gmjp"
server_key: "902762a912a55e06f82ddc3a49c4333b"
server_key_iv: "1773f3c00e0f89b69cb0d9c15f174cd2"
server_key_version: "encryptionkey@0"
I20260806 02:45:33.555655 6213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "a8c6610947fb49239893ee32b0d313a7"
format_stamp: "Formatted at 2026-08-06 02:45:33 on dist-test-slave-gmjp"
server_key: "902762a912a55e06f82ddc3a49c4333b"
server_key_iv: "1773f3c00e0f89b69cb0d9c15f174cd2"
server_key_version: "encryptionkey@0"
I20260806 02:45:33.559728 6213 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:45:33.562453 6235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:33.563745 6213 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:45:33.563905 6213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "a8c6610947fb49239893ee32b0d313a7"
format_stamp: "Formatted at 2026-08-06 02:45:33 on dist-test-slave-gmjp"
server_key: "902762a912a55e06f82ddc3a49c4333b"
server_key_iv: "1773f3c00e0f89b69cb0d9c15f174cd2"
server_key_version: "encryptionkey@0"
I20260806 02:45:33.564039 6213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:33.578822 6213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:33.582289 6213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:33.582535 6213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:33.590386 6213 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:46457
I20260806 02:45:33.590704 6317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:46457 every 8 connection(s)
I20260806 02:45:33.591521 6213 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:33.594871 6319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:33.599750 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 6213
I20260806 02:45:33.599905 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:45:33.600258 1758 external_mini_cluster.cc:1428] Setting key ba0d4883388f742cd207f61063ee1911
I20260806 02:45:33.601861 6319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8c6610947fb49239893ee32b0d313a7: Bootstrap starting.
I20260806 02:45:33.604519 6319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8c6610947fb49239893ee32b0d313a7: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:33.605386 6319 log.cc:824] T 00000000000000000000000000000000 P a8c6610947fb49239893ee32b0d313a7: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:33.607620 6319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8c6610947fb49239893ee32b0d313a7: No bootstrap required, opened a new log
Aug 06 02:45:33 dist-test-slave-gmjp krb5kdc[6189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:33.612534 6319 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8c6610947fb49239893ee32b0d313a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c6610947fb49239893ee32b0d313a7" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 46457 } }
I20260806 02:45:33.612896 6319 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8c6610947fb49239893ee32b0d313a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:33.612978 6319 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8c6610947fb49239893ee32b0d313a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c6610947fb49239893ee32b0d313a7, State: Initialized, Role: FOLLOWER
I20260806 02:45:33.613651 6319 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8c6610947fb49239893ee32b0d313a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c6610947fb49239893ee32b0d313a7" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 46457 } }
I20260806 02:45:33.613857 6319 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8c6610947fb49239893ee32b0d313a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:33.613960 6319 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8c6610947fb49239893ee32b0d313a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:33.614076 6319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8c6610947fb49239893ee32b0d313a7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:33.615486 6319 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8c6610947fb49239893ee32b0d313a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c6610947fb49239893ee32b0d313a7" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 46457 } }
I20260806 02:45:33.615942 6319 leader_election.cc:302] T 00000000000000000000000000000000 P a8c6610947fb49239893ee32b0d313a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8c6610947fb49239893ee32b0d313a7; no voters:
I20260806 02:45:33.616279 6323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:33.601853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44840 (local address 127.1.183.190:46457)
0806 02:45:33.602338 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:33.602353 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:33.602390 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:45:33.603053 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:33.603070 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:33.603434 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:33.603791 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:33.604781 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:33.605833 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:33.606733 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:33.607216 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:33.607398 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:33.610524 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:33.610530 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:33.610547 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:33.610585 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:33.612608 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:33.613238 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:33.613243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:33.613249 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:33.613373 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:33.613705 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:33.613707 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:33.613708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:33.614311 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:33.614486 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:33.614838 (+ 352us) server_negotiation.cc:299] Negotiation successful
0806 02:45:33.615188 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":155,"threads_started":1}
I20260806 02:45:33.616353 6319 leader_election.cc:288] T 00000000000000000000000000000000 P a8c6610947fb49239893ee32b0d313a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:33.616600 6325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8c6610947fb49239893ee32b0d313a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:33.616845 6325 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8c6610947fb49239893ee32b0d313a7 [term 1 LEADER]: Becoming Leader. State: Replica: a8c6610947fb49239893ee32b0d313a7, State: Running, Role: LEADER
I20260806 02:45:33.617255 6325 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8c6610947fb49239893ee32b0d313a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c6610947fb49239893ee32b0d313a7" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 46457 } }
I20260806 02:45:33.618026 6319 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8c6610947fb49239893ee32b0d313a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:33.618588 6325 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8c6610947fb49239893ee32b0d313a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8c6610947fb49239893ee32b0d313a7. Latest consensus state: current_term: 1 leader_uuid: "a8c6610947fb49239893ee32b0d313a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c6610947fb49239893ee32b0d313a7" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 46457 } } }
I20260806 02:45:33.618731 6325 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8c6610947fb49239893ee32b0d313a7 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:33.619503 6332 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8c6610947fb49239893ee32b0d313a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8c6610947fb49239893ee32b0d313a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c6610947fb49239893ee32b0d313a7" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 46457 } } }
I20260806 02:45:33.619611 6332 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8c6610947fb49239893ee32b0d313a7 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:33.619871 6336 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:33.620213 1758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:45:33.623297 6336 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:33.629204 6336 catalog_manager.cc:1347] Generated new cluster ID: cf2ade598b4a4bf5983d8cd74c11b612
I20260806 02:45:33.629285 6336 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:33.637706 6336 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:33.638765 6336 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:33.647087 6336 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8c6610947fb49239893ee32b0d313a7: Generated new TSK 0
I20260806 02:45:33.647938 6336 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:45:33.648748 6323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:33.642022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44852 (local address 127.1.183.190:46457)
0806 02:45:33.642166 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:33.642172 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:33.642496 (+ 324us) server_negotiation.cc:407] Connection header received
0806 02:45:33.642646 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:33.642650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:33.642700 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:33.642795 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:33.643525 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:33.644422 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:33.645856 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:33.646166 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:33.646292 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:33.647486 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:33.647490 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:33.647494 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:33.648174 (+ 680us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:33.648217 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:33.648252 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:33.648356 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:33.648507 (+ 151us) server_negotiation.cc:299] Negotiation successful
0806 02:45:33.648588 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:45:33.654578 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 6213
2026-08-06T02:45:33Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1646 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:45:33 dist-test-slave-gmjp krb5kdc[6374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:45:33 dist-test-slave-gmjp krb5kdc[6374](info): set up 2 sockets
Aug 06 02:45:33 dist-test-slave-gmjp krb5kdc[6374](info): commencing operation
krb5kdc: starting...
W20260806 02:45:34.883868 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.202s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:45:34 dist-test-slave-gmjp krb5kdc[6374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:45:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
I20260806 02:45:35.023020 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:38853
--webserver_interface=127.1.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:43001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:38853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:35.141894 6393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:35.142266 6393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:35.142354 6393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:35.146621 6393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:35.146701 6393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:35.146721 6393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:35.146742 6393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:35.146759 6393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:35.152559 6393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:43001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:38853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:38853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:35.153818 6393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:35.155004 6393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:35.164610 6400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:35.164588 6402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:35.164587 6398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:35.164788 6393 server_base.cc:1034] running on GCE node
I20260806 02:45:35.165686 6393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:35.166975 6393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:35.168238 6393 hybrid_clock.cc:648] HybridClock initialized: now 1785984335168222 us; error 33 us; skew 500 ppm
Aug 06 02:45:35 dist-test-slave-gmjp krb5kdc[6374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984335, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:35.174748 6393 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:45:35.175894 6393 webserver.cc:460] Webserver started at http://127.1.183.190:42135/ using document root <none> and password file <none>
I20260806 02:45:35.176585 6393 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:35.176664 6393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:35.176887 6393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:35.179177 6393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "1ec95a7aed9c41949c6a45dbdf8e106f"
format_stamp: "Formatted at 2026-08-06 02:45:35 on dist-test-slave-gmjp"
server_key: "dacc4625021895684526d2f615b2226b"
server_key_iv: "228ee17262c40e6611280a144553c2c9"
server_key_version: "encryptionkey@0"
I20260806 02:45:35.179756 6393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "1ec95a7aed9c41949c6a45dbdf8e106f"
format_stamp: "Formatted at 2026-08-06 02:45:35 on dist-test-slave-gmjp"
server_key: "dacc4625021895684526d2f615b2226b"
server_key_iv: "228ee17262c40e6611280a144553c2c9"
server_key_version: "encryptionkey@0"
I20260806 02:45:35.185289 6393 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 02:45:35.188941 6412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:35.190696 6393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:45:35.190860 6393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "1ec95a7aed9c41949c6a45dbdf8e106f"
format_stamp: "Formatted at 2026-08-06 02:45:35 on dist-test-slave-gmjp"
server_key: "dacc4625021895684526d2f615b2226b"
server_key_iv: "228ee17262c40e6611280a144553c2c9"
server_key_version: "encryptionkey@0"
I20260806 02:45:35.191011 6393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:35.210662 6393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:35.214010 6393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:35.214247 6393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:35.222252 6393 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:38853
I20260806 02:45:35.222572 6493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:38853 every 8 connection(s)
I20260806 02:45:35.223470 6393 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:35.226647 6497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:35.231707 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 6393
I20260806 02:45:35.231863 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:45:35.232205 1758 external_mini_cluster.cc:1428] Setting key f0e66c0f2832bf426f0cf8dc3f980841
I20260806 02:45:35.234046 6497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ec95a7aed9c41949c6a45dbdf8e106f: Bootstrap starting.
I20260806 02:45:35.236769 6497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ec95a7aed9c41949c6a45dbdf8e106f: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:35.237648 6497 log.cc:824] T 00000000000000000000000000000000 P 1ec95a7aed9c41949c6a45dbdf8e106f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:35.240335 6497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ec95a7aed9c41949c6a45dbdf8e106f: No bootstrap required, opened a new log
Aug 06 02:45:35 dist-test-slave-gmjp krb5kdc[6374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:35.244434 6497 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ec95a7aed9c41949c6a45dbdf8e106f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec95a7aed9c41949c6a45dbdf8e106f" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38853 } }
I20260806 02:45:35.244733 6497 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ec95a7aed9c41949c6a45dbdf8e106f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:35.244822 6497 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ec95a7aed9c41949c6a45dbdf8e106f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ec95a7aed9c41949c6a45dbdf8e106f, State: Initialized, Role: FOLLOWER
I20260806 02:45:35.245317 6497 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ec95a7aed9c41949c6a45dbdf8e106f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec95a7aed9c41949c6a45dbdf8e106f" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38853 } }
I20260806 02:45:35.245481 6497 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ec95a7aed9c41949c6a45dbdf8e106f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:35.245564 6497 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ec95a7aed9c41949c6a45dbdf8e106f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:35.245693 6497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ec95a7aed9c41949c6a45dbdf8e106f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:35.246713 6497 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ec95a7aed9c41949c6a45dbdf8e106f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec95a7aed9c41949c6a45dbdf8e106f" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38853 } }
I20260806 02:45:35.247085 6497 leader_election.cc:302] T 00000000000000000000000000000000 P 1ec95a7aed9c41949c6a45dbdf8e106f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ec95a7aed9c41949c6a45dbdf8e106f; no voters:
I20260806 02:45:35.247440 6497 leader_election.cc:288] T 00000000000000000000000000000000 P 1ec95a7aed9c41949c6a45dbdf8e106f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:35.247577 6502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ec95a7aed9c41949c6a45dbdf8e106f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:35.247849 6502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ec95a7aed9c41949c6a45dbdf8e106f [term 1 LEADER]: Becoming Leader. State: Replica: 1ec95a7aed9c41949c6a45dbdf8e106f, State: Running, Role: LEADER
I20260806 02:45:35.248225 6502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ec95a7aed9c41949c6a45dbdf8e106f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec95a7aed9c41949c6a45dbdf8e106f" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38853 } }
I20260806 02:45:35.248952 6497 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ec95a7aed9c41949c6a45dbdf8e106f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:35.249348 6500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:35.233959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54626 (local address 127.1.183.190:38853)
0806 02:45:35.234478 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:35.234494 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:35.234533 (+ 39us) server_negotiation.cc:407] Connection header received
0806 02:45:35.235231 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:35.235250 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:35.235617 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:35.236124 (+ 507us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:35.237177 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:35.238380 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:35.239726 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:35.240352 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:35.240550 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:35.243241 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:35.243248 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:35.243264 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:35.243297 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:35.245352 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:35.245930 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:35.245944 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:35.245949 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:35.246070 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:35.246393 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:35.246401 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:35.246403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:35.246877 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:35.247102 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:35.247945 (+ 843us) server_negotiation.cc:299] Negotiation successful
0806 02:45:35.248320 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":157,"threads_started":1}
I20260806 02:45:35.249805 6505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ec95a7aed9c41949c6a45dbdf8e106f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ec95a7aed9c41949c6a45dbdf8e106f. Latest consensus state: current_term: 1 leader_uuid: "1ec95a7aed9c41949c6a45dbdf8e106f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec95a7aed9c41949c6a45dbdf8e106f" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38853 } } }
I20260806 02:45:35.249954 6505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ec95a7aed9c41949c6a45dbdf8e106f [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:35.250036 6504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ec95a7aed9c41949c6a45dbdf8e106f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ec95a7aed9c41949c6a45dbdf8e106f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ec95a7aed9c41949c6a45dbdf8e106f" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38853 } } }
I20260806 02:45:35.250124 6504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ec95a7aed9c41949c6a45dbdf8e106f [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:35.250373 6512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:35.253993 6512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:35.261332 6512 catalog_manager.cc:1347] Generated new cluster ID: b9b9152e4c6a4431be8be589b782f210
I20260806 02:45:35.261445 6512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:35.275759 6512 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:35.276870 6512 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:35.282111 6512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ec95a7aed9c41949c6a45dbdf8e106f: Generated new TSK 0
I20260806 02:45:35.282882 6512 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
I20260806 02:45:35.327310 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.129:0
--local_ip_for_outbound_sockets=127.1.183.129
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:38853
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:43001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:35.452924 6536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:35.453281 6536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:35.453364 6536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:35.457770 6536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:35.457887 6536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:35.457995 6536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.129
I20260806 02:45:35.463698 6536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:43001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:38853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.129
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:35.465075 6536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:35.466239 6536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:45:35.476926 6536 server_base.cc:1034] running on GCE node
W20260806 02:45:35.476871 6542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:35.476878 6547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:35.476878 6544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:35.477656 6536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:35.478488 6536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:35.479708 6536 hybrid_clock.cc:648] HybridClock initialized: now 1785984335479692 us; error 35 us; skew 500 ppm
Aug 06 02:45:35 dist-test-slave-gmjp krb5kdc[6374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984335, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:35.483140 6536 init.cc:377] Logged in from keytab as kudu/127.1.183.129@KRBTEST.COM (short username kudu)
I20260806 02:45:35.484301 6536 webserver.cc:460] Webserver started at http://127.1.183.129:46749/ using document root <none> and password file <none>
I20260806 02:45:35.484943 6536 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:35.485060 6536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:35.485285 6536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:35.487536 6536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/data/instance:
uuid: "8a9ab59c514a4f909c3e72a85a4ba064"
format_stamp: "Formatted at 2026-08-06 02:45:35 on dist-test-slave-gmjp"
server_key: "f450bdce1a423b7b8eb1ef84997e5b4a"
server_key_iv: "3718bef0250ad64e752771820be1ee4b"
server_key_version: "encryptionkey@0"
I20260806 02:45:35.488135 6536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance:
uuid: "8a9ab59c514a4f909c3e72a85a4ba064"
format_stamp: "Formatted at 2026-08-06 02:45:35 on dist-test-slave-gmjp"
server_key: "f450bdce1a423b7b8eb1ef84997e5b4a"
server_key_iv: "3718bef0250ad64e752771820be1ee4b"
server_key_version: "encryptionkey@0"
I20260806 02:45:35.492551 6536 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260806 02:45:35.495370 6557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:35.496995 6536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:35.497144 6536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/wal
uuid: "8a9ab59c514a4f909c3e72a85a4ba064"
format_stamp: "Formatted at 2026-08-06 02:45:35 on dist-test-slave-gmjp"
server_key: "f450bdce1a423b7b8eb1ef84997e5b4a"
server_key_iv: "3718bef0250ad64e752771820be1ee4b"
server_key_version: "encryptionkey@0"
I20260806 02:45:35.497282 6536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:35.516088 6536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:35.519971 6536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:35.520251 6536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:35.520959 6536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:35.521873 6536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:35.521924 6536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:35.521996 6536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:35.522027 6536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:35.531841 6536 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.129:46479
I20260806 02:45:35.531884 6734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.129:46479 every 8 connection(s)
I20260806 02:45:35.532884 6536 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
I20260806 02:45:35.536137 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 6536
I20260806 02:45:35.536309 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance
I20260806 02:45:35.536667 1758 external_mini_cluster.cc:1428] Setting key de7a97e430681151a49bc5aeb3547160
Aug 06 02:45:35 dist-test-slave-gmjp krb5kdc[6374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984335, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.183.130@KRBTEST.COM; defaulting to no policy
I20260806 02:45:35.549986 6500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:35.534964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:33717 (local address 127.1.183.190:38853)
0806 02:45:35.535136 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:35.535141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:35.536338 (+ 1197us) server_negotiation.cc:407] Connection header received
0806 02:45:35.538092 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:35.538100 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:35.538169 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:35.538289 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:35.540093 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:35.540842 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:35.542035 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:35.542439 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:35.542573 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:35.545615 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:35.545619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:35.545623 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:35.545661 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:35.547431 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:35.547987 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:35.547993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:35.547998 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:35.548097 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:35.548632 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:35.548637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:35.548642 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:35.548891 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:35.548983 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:35.549628 (+ 645us) server_negotiation.cc:299] Negotiation successful
0806 02:45:35.549846 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 02:45:35.551033 6739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:35.535422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:38853 (local address 127.1.183.129:33717)
0806 02:45:35.536015 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:35.536096 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:35.537854 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:35.538518 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:35.538529 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:35.539104 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:35.539882 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:35.539915 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:35.540994 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:35.540997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:35.541886 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:35.541896 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:35.542602 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:35.542604 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:35.542760 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:35.543439 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:35.543457 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:35.545396 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:35.547566 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:35.547571 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:35.547588 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:35.547870 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:35.548242 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:35.548245 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:35.548247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:35.548461 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:35.548994 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:35.549000 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:35.549277 (+ 277us) client_negotiation.cc:769] Sending connection context
0806 02:45:35.549599 (+ 322us) client_negotiation.cc:240] Negotiation successful
0806 02:45:35.549953 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.130@KRBTEST.COM" created.
I20260806 02:45:35.552577 6735 heartbeater.cc:347] Connected to a master server at 127.1.183.190:38853
I20260806 02:45:35.552879 6735 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:35.553565 6735 heartbeater.cc:511] Master 127.1.183.190:38853 requested a full tablet report, sending...
I20260806 02:45:35.555289 6437 ts_manager.cc:194] Registered new tserver with Master: 8a9ab59c514a4f909c3e72a85a4ba064 (127.1.183.129:46479)
I20260806 02:45:35.556569 6437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.129@KRBTEST.COM'} at 127.1.183.129:33717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
I20260806 02:45:35.568797 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.130:0
--local_ip_for_outbound_sockets=127.1.183.130
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:38853
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:43001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:35.691020 6744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:35.691327 6744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:35.691375 6744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:35.696039 6744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:35.696116 6744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:35.696256 6744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.130
I20260806 02:45:35.702488 6744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:43001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:38853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.130
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:35.703725 6744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:35.704900 6744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:45:35.715462 6744 server_base.cc:1034] running on GCE node
W20260806 02:45:35.715409 6753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:35.715389 6750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:35.715392 6751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:35.716114 6744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:35.716931 6744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:35.718119 6744 hybrid_clock.cc:648] HybridClock initialized: now 1785984335718101 us; error 30 us; skew 500 ppm
Aug 06 02:45:35 dist-test-slave-gmjp krb5kdc[6374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984335, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:35.721258 6744 init.cc:377] Logged in from keytab as kudu/127.1.183.130@KRBTEST.COM (short username kudu)
I20260806 02:45:35.722209 6744 webserver.cc:460] Webserver started at http://127.1.183.130:32903/ using document root <none> and password file <none>
I20260806 02:45:35.722784 6744 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:35.722848 6744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:35.723037 6744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:35.725332 6744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/data/instance:
uuid: "4d16e53cd0c2415a85307ffcf18505d3"
format_stamp: "Formatted at 2026-08-06 02:45:35 on dist-test-slave-gmjp"
server_key: "fdf4e182728f4f018276e17a31c0cfb8"
server_key_iv: "59a2e9ef1eece59fa14ee59d8235cf5c"
server_key_version: "encryptionkey@0"
I20260806 02:45:35.725905 6744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance:
uuid: "4d16e53cd0c2415a85307ffcf18505d3"
format_stamp: "Formatted at 2026-08-06 02:45:35 on dist-test-slave-gmjp"
server_key: "fdf4e182728f4f018276e17a31c0cfb8"
server_key_iv: "59a2e9ef1eece59fa14ee59d8235cf5c"
server_key_version: "encryptionkey@0"
I20260806 02:45:35.730104 6744 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:45:35.732847 6762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:35.734272 6744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:35.734397 6744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/wal
uuid: "4d16e53cd0c2415a85307ffcf18505d3"
format_stamp: "Formatted at 2026-08-06 02:45:35 on dist-test-slave-gmjp"
server_key: "fdf4e182728f4f018276e17a31c0cfb8"
server_key_iv: "59a2e9ef1eece59fa14ee59d8235cf5c"
server_key_version: "encryptionkey@0"
I20260806 02:45:35.734503 6744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:35.782982 6744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:35.786733 6744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:35.786929 6744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:35.787554 6744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:35.788597 6744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:35.788651 6744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:35.788694 6744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:35.788710 6744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:35.798775 6744 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.130:44901
I20260806 02:45:35.798856 6944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.130:44901 every 8 connection(s)
I20260806 02:45:35.799787 6744 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
I20260806 02:45:35.807904 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 6744
I20260806 02:45:35.808048 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance
I20260806 02:45:35.808441 1758 external_mini_cluster.cc:1428] Setting key d7decba858a5652ba85ccb501beae592
Aug 06 02:45:35 dist-test-slave-gmjp krb5kdc[6374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984335, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:35.816517 6500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:35.802057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:54981 (local address 127.1.183.190:38853)
0806 02:45:35.802215 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:35.802219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:35.803165 (+ 946us) server_negotiation.cc:407] Connection header received
0806 02:45:35.804685 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:35.804689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:35.804742 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:35.804834 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:35.806095 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:35.806637 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:35.807867 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:35.808269 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:35.808411 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:35.811854 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:35.811857 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:35.811859 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:35.811886 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:35.813850 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:35.814539 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:35.814546 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:35.814551 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:35.814653 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:35.815083 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:35.815089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:35.815093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:35.815381 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:35.815494 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:35.816117 (+ 623us) server_negotiation.cc:299] Negotiation successful
0806 02:45:35.816309 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 02:45:35.817447 6948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:35.802478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:38853 (local address 127.1.183.130:54981)
0806 02:45:35.802984 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:35.803031 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:35.804456 (+ 1425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:35.805012 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:35.805019 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:35.805435 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:35.805947 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:35.805962 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:35.806759 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:35.806762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:35.807715 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:35.807725 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:35.808458 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:35.808461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:35.808678 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:35.809392 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:35.809411 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:35.811627 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:35.814011 (+ 2384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:35.814019 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:35.814037 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:35.814409 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:35.814784 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:35.814788 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:35.814789 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:35.814971 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:35.815495 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:35.815501 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:35.815777 (+ 276us) client_negotiation.cc:769] Sending connection context
0806 02:45:35.816031 (+ 254us) client_negotiation.cc:240] Negotiation successful
0806 02:45:35.816435 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":128,"threads_started":1}
I20260806 02:45:35.818912 6946 heartbeater.cc:347] Connected to a master server at 127.1.183.190:38853
I20260806 02:45:35.819231 6946 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:35.819798 6946 heartbeater.cc:511] Master 127.1.183.190:38853 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.183.131@KRBTEST.COM; defaulting to no policy
I20260806 02:45:35.821161 6437 ts_manager.cc:194] Registered new tserver with Master: 4d16e53cd0c2415a85307ffcf18505d3 (127.1.183.130:44901)
I20260806 02:45:35.821825 6437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.130@KRBTEST.COM'} at 127.1.183.130:54981
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
I20260806 02:45:35.843043 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.131:0
--local_ip_for_outbound_sockets=127.1.183.131
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:38853
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:43001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:35.962997 6952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:35.963297 6952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:35.963343 6952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:35.967459 6952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:35.967532 6952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:35.967595 6952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.131
I20260806 02:45:35.973809 6952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:43001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:38853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.131
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:35.975167 6952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:35.976295 6952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:35.986872 6960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:35.986886 6963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:35.986847 6959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:35.987486 6952 server_base.cc:1034] running on GCE node
I20260806 02:45:35.987975 6952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:35.988762 6952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:35.989964 6952 hybrid_clock.cc:648] HybridClock initialized: now 1785984335989944 us; error 35 us; skew 500 ppm
Aug 06 02:45:35 dist-test-slave-gmjp krb5kdc[6374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984335, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:35.993037 6952 init.cc:377] Logged in from keytab as kudu/127.1.183.131@KRBTEST.COM (short username kudu)
I20260806 02:45:35.994089 6952 webserver.cc:460] Webserver started at http://127.1.183.131:37197/ using document root <none> and password file <none>
I20260806 02:45:35.994776 6952 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:35.994874 6952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:35.995127 6952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:35.997435 6952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/data/instance:
uuid: "93ecce7ee9634890b015b405a30852c1"
format_stamp: "Formatted at 2026-08-06 02:45:35 on dist-test-slave-gmjp"
server_key: "c45e95d148a3c5695b53897dc5d31127"
server_key_iv: "305c59d733374081deaf5a850c9a7984"
server_key_version: "encryptionkey@0"
I20260806 02:45:35.998046 6952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance:
uuid: "93ecce7ee9634890b015b405a30852c1"
format_stamp: "Formatted at 2026-08-06 02:45:35 on dist-test-slave-gmjp"
server_key: "c45e95d148a3c5695b53897dc5d31127"
server_key_iv: "305c59d733374081deaf5a850c9a7984"
server_key_version: "encryptionkey@0"
I20260806 02:45:36.002403 6952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:45:36.005210 6972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:36.006623 6952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:36.006780 6952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/wal
uuid: "93ecce7ee9634890b015b405a30852c1"
format_stamp: "Formatted at 2026-08-06 02:45:35 on dist-test-slave-gmjp"
server_key: "c45e95d148a3c5695b53897dc5d31127"
server_key_iv: "305c59d733374081deaf5a850c9a7984"
server_key_version: "encryptionkey@0"
I20260806 02:45:36.006912 6952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:36.027763 6952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:36.031155 6952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:36.031418 6952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:36.032099 6952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:36.033062 6952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:36.033141 6952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:36.033218 6952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:36.033262 6952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:36.043107 6952 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.131:37779
I20260806 02:45:36.043129 7151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.131:37779 every 8 connection(s)
I20260806 02:45:36.044138 6952 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
I20260806 02:45:36.051527 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 6952
I20260806 02:45:36.051645 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance
I20260806 02:45:36.051970 1758 external_mini_cluster.cc:1428] Setting key ee74bffb6289ef437179a357eff93b0d
Aug 06 02:45:36 dist-test-slave-gmjp krb5kdc[6374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984335, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:36.059939 6500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:36.046189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:37383 (local address 127.1.183.190:38853)
0806 02:45:36.046362 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:36.046367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:36.047367 (+ 1000us) server_negotiation.cc:407] Connection header received
0806 02:45:36.048967 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:36.048970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:36.049022 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:36.049101 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:36.050522 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:36.051131 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:36.052728 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:36.052996 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:36.053119 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:36.055891 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:36.055894 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:36.055897 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:36.055926 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:36.057569 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:36.058231 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:36.058234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:36.058236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:36.058324 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:36.058713 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:36.058716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:36.058717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:36.058929 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:36.059022 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:36.059663 (+ 641us) server_negotiation.cc:299] Negotiation successful
0806 02:45:36.059801 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 02:45:36.060938 7154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:36.046656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:38853 (local address 127.1.183.131:37383)
0806 02:45:36.047176 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:36.047225 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:36.048656 (+ 1431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:36.049291 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:36.049298 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:36.049710 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:36.050373 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:36.050388 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:36.051288 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:36.051291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:36.052571 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:36.052584 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:36.053161 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:36.053164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:36.053352 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:36.054060 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:36.054077 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:36.055716 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:36.057716 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:36.057723 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:36.057740 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:36.058099 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:36.058431 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:36.058433 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:36.058435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:36.058598 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:36.059072 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:36.059079 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:36.059389 (+ 310us) client_negotiation.cc:769] Sending connection context
0806 02:45:36.059582 (+ 193us) client_negotiation.cc:240] Negotiation successful
0806 02:45:36.059943 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":126,"threads_started":1}
I20260806 02:45:36.062317 7152 heartbeater.cc:347] Connected to a master server at 127.1.183.190:38853
I20260806 02:45:36.062600 7152 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:36.063198 7152 heartbeater.cc:511] Master 127.1.183.190:38853 requested a full tablet report, sending...
I20260806 02:45:36.064596 6437 ts_manager.cc:194] Registered new tserver with Master: 93ecce7ee9634890b015b405a30852c1 (127.1.183.131:37779)
I20260806 02:45:36.065330 6437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.131@KRBTEST.COM'} at 127.1.183.131:37383
I20260806 02:45:36.066504 1758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:45:36 dist-test-slave-gmjp krb5kdc[6374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984336, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 06 02:45:36 dist-test-slave-gmjp krb5kdc[6374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984336, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:36.095579 6500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:36.086269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54628 (local address 127.1.183.190:38853)
0806 02:45:36.086438 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:36.086442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:36.086707 (+ 265us) server_negotiation.cc:407] Connection header received
0806 02:45:36.086855 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:36.086858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:36.086907 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:36.086991 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:36.087830 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:36.088485 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:36.089359 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:36.089573 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:36.089655 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:36.092249 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:36.092253 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:36.092255 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:36.092286 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:36.093925 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:36.094311 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:36.094313 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:36.094315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:36.094423 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:36.094654 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:36.094656 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:36.094657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:36.094935 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:36.095047 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:36.095300 (+ 253us) server_negotiation.cc:299] Negotiation successful
0806 02:45:36.095406 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260806 02:45:36.096050 6437 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54628
I20260806 02:45:36.097069 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 6536
I20260806 02:45:36.110874 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 6744
I20260806 02:45:36.123508 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 6952
I20260806 02:45:36.140762 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 6393
2026-08-06T02:45:36Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2490 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:45:36 dist-test-slave-gmjp krb5kdc[7169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:45:36 dist-test-slave-gmjp krb5kdc[7169](info): set up 2 sockets
Aug 06 02:45:36 dist-test-slave-gmjp krb5kdc[7169](info): commencing operation
krb5kdc: starting...
W20260806 02:45:37.407734 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:45:37 dist-test-slave-gmjp krb5kdc[7169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:45:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
I20260806 02:45:37.544427 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:37317
--webserver_interface=127.1.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:45879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:37317
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:37.705687 7197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:37.706182 7197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:37.706260 7197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:37.714867 7197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:37.714993 7197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:37.715034 7197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:37.715065 7197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:37.715093 7197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:37.726351 7197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:45879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:37317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:37317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:37.728078 7197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:37.729687 7197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:37.741176 7213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:37.741196 7211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:37.741250 7210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:37.742462 7197 server_base.cc:1034] running on GCE node
I20260806 02:45:37.743323 7197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:37.745126 7197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:37.746346 7197 hybrid_clock.cc:648] HybridClock initialized: now 1785984337746282 us; error 88 us; skew 500 ppm
Aug 06 02:45:37 dist-test-slave-gmjp krb5kdc[7169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984337, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:37.750183 7197 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:45:37.751672 7197 webserver.cc:460] Webserver started at http://127.1.183.190:41559/ using document root <none> and password file <none>
I20260806 02:45:37.752607 7197 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:37.752727 7197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:37.753039 7197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:37.755251 7197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "a0d8243023d14bb9a7c89696412cc270"
format_stamp: "Formatted at 2026-08-06 02:45:37 on dist-test-slave-gmjp"
server_key: "efd3d232208e6009694b49c07cb6e725"
server_key_iv: "0207ae81336b42bae963e2a367a7c761"
server_key_version: "encryptionkey@0"
I20260806 02:45:37.755795 7197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "a0d8243023d14bb9a7c89696412cc270"
format_stamp: "Formatted at 2026-08-06 02:45:37 on dist-test-slave-gmjp"
server_key: "efd3d232208e6009694b49c07cb6e725"
server_key_iv: "0207ae81336b42bae963e2a367a7c761"
server_key_version: "encryptionkey@0"
I20260806 02:45:37.759994 7197 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 02:45:37.762751 7228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:37.764248 7197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:37.764410 7197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "a0d8243023d14bb9a7c89696412cc270"
format_stamp: "Formatted at 2026-08-06 02:45:37 on dist-test-slave-gmjp"
server_key: "efd3d232208e6009694b49c07cb6e725"
server_key_iv: "0207ae81336b42bae963e2a367a7c761"
server_key_version: "encryptionkey@0"
I20260806 02:45:37.764540 7197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:37.783246 7197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:37.786902 7197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:37.787168 7197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:37.794920 7197 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:37317
I20260806 02:45:37.794950 7306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:37317 every 8 connection(s)
I20260806 02:45:37.796033 7197 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:37.799338 7307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:37.803639 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 7197
I20260806 02:45:37.803821 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:45:37.804118 1758 external_mini_cluster.cc:1428] Setting key c5f9f8180aa44a23436163ea569ccd0f
I20260806 02:45:37.805894 7307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0d8243023d14bb9a7c89696412cc270: Bootstrap starting.
I20260806 02:45:37.808912 7307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0d8243023d14bb9a7c89696412cc270: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:37.809753 7307 log.cc:824] T 00000000000000000000000000000000 P a0d8243023d14bb9a7c89696412cc270: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:37.811928 7307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0d8243023d14bb9a7c89696412cc270: No bootstrap required, opened a new log
Aug 06 02:45:37 dist-test-slave-gmjp krb5kdc[7169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:37.816084 7307 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0d8243023d14bb9a7c89696412cc270 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0d8243023d14bb9a7c89696412cc270" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 37317 } }
I20260806 02:45:37.816425 7307 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0d8243023d14bb9a7c89696412cc270 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:37.816514 7307 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0d8243023d14bb9a7c89696412cc270 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d8243023d14bb9a7c89696412cc270, State: Initialized, Role: FOLLOWER
I20260806 02:45:37.817013 7307 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0d8243023d14bb9a7c89696412cc270 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0d8243023d14bb9a7c89696412cc270" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 37317 } }
I20260806 02:45:37.817166 7307 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0d8243023d14bb9a7c89696412cc270 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:37.817240 7307 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0d8243023d14bb9a7c89696412cc270 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:37.817342 7307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0d8243023d14bb9a7c89696412cc270 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:37.818317 7307 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0d8243023d14bb9a7c89696412cc270 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0d8243023d14bb9a7c89696412cc270" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 37317 } }
I20260806 02:45:37.818707 7307 leader_election.cc:302] T 00000000000000000000000000000000 P a0d8243023d14bb9a7c89696412cc270 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0d8243023d14bb9a7c89696412cc270; no voters:
I20260806 02:45:37.819085 7307 leader_election.cc:288] T 00000000000000000000000000000000 P a0d8243023d14bb9a7c89696412cc270 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:37.819252 7316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0d8243023d14bb9a7c89696412cc270 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:37.819489 7316 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0d8243023d14bb9a7c89696412cc270 [term 1 LEADER]: Becoming Leader. State: Replica: a0d8243023d14bb9a7c89696412cc270, State: Running, Role: LEADER
I20260806 02:45:37.819866 7316 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0d8243023d14bb9a7c89696412cc270 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0d8243023d14bb9a7c89696412cc270" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 37317 } }
I20260806 02:45:37.820633 7307 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0d8243023d14bb9a7c89696412cc270 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:37.820650 7312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:37.805826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52212 (local address 127.1.183.190:37317)
0806 02:45:37.806357 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:37.806377 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:37.806423 (+ 46us) server_negotiation.cc:407] Connection header received
0806 02:45:37.807354 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:37.807384 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:37.807828 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:37.808265 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:37.809152 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:37.810249 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:37.811163 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:37.811672 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:37.811871 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:37.814824 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:37.814830 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:37.814847 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:37.814880 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:37.816963 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:37.817594 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:37.817600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:37.817605 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:37.817739 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:37.818068 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:37.818072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:37.818074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:37.818612 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:37.818841 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:37.819181 (+ 340us) server_negotiation.cc:299] Negotiation successful
0806 02:45:37.819583 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":186,"threads_started":1}
I20260806 02:45:37.821798 7317 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0d8243023d14bb9a7c89696412cc270 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0d8243023d14bb9a7c89696412cc270" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0d8243023d14bb9a7c89696412cc270" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 37317 } } }
I20260806 02:45:37.822263 7317 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0d8243023d14bb9a7c89696412cc270 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:37.822139 7318 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0d8243023d14bb9a7c89696412cc270 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0d8243023d14bb9a7c89696412cc270. Latest consensus state: current_term: 1 leader_uuid: "a0d8243023d14bb9a7c89696412cc270" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0d8243023d14bb9a7c89696412cc270" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 37317 } } }
I20260806 02:45:37.822630 7318 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0d8243023d14bb9a7c89696412cc270 [sys.catalog]: This master's current role is: LEADER
W20260806 02:45:37.826225 7336 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a0d8243023d14bb9a7c89696412cc270: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:45:37.826305 7336 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:45:37.826424 7331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:37.827133 7331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:37.832949 7331 catalog_manager.cc:1347] Generated new cluster ID: ee88f5aeb7c4417689ba21aaf73fe534
I20260806 02:45:37.833035 7331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:37.853168 7331 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:37.854219 7331 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:37.862586 7331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0d8243023d14bb9a7c89696412cc270: Generated new TSK 0
I20260806 02:45:37.863404 7331 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
I20260806 02:45:37.899571 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.129:0
--local_ip_for_outbound_sockets=127.1.183.129
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:37317
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:45879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:38.023254 7344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:38.023563 7344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:38.023612 7344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:38.028115 7344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:38.028251 7344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:38.028318 7344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.129
I20260806 02:45:38.034001 7344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:45879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:37317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.129
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:38.035198 7344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:38.036393 7344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:45:38.046782 7344 server_base.cc:1034] running on GCE node
W20260806 02:45:38.046792 7357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:38.046814 7358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:38.046819 7361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:38.047425 7344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:38.048112 7344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:38.049383 7344 hybrid_clock.cc:648] HybridClock initialized: now 1785984338049334 us; error 52 us; skew 500 ppm
Aug 06 02:45:38 dist-test-slave-gmjp krb5kdc[7169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984338, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:38.052424 7344 init.cc:377] Logged in from keytab as kudu/127.1.183.129@KRBTEST.COM (short username kudu)
I20260806 02:45:38.053328 7344 webserver.cc:460] Webserver started at http://127.1.183.129:41609/ using document root <none> and password file <none>
I20260806 02:45:38.053874 7344 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:38.053937 7344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:38.054114 7344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:38.056334 7344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-0/data/instance:
uuid: "e628ed6a84df4cbfa77a81cee9703718"
format_stamp: "Formatted at 2026-08-06 02:45:38 on dist-test-slave-gmjp"
server_key: "5a44f975c9cbba4779daf39a7a476187"
server_key_iv: "d8ceb6d3124947318f7c847b65e9f57b"
server_key_version: "encryptionkey@0"
I20260806 02:45:38.056918 7344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance:
uuid: "e628ed6a84df4cbfa77a81cee9703718"
format_stamp: "Formatted at 2026-08-06 02:45:38 on dist-test-slave-gmjp"
server_key: "5a44f975c9cbba4779daf39a7a476187"
server_key_iv: "d8ceb6d3124947318f7c847b65e9f57b"
server_key_version: "encryptionkey@0"
I20260806 02:45:38.061039 7344 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:45:38.063650 7370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:38.065021 7344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:45:38.065177 7344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-0/wal
uuid: "e628ed6a84df4cbfa77a81cee9703718"
format_stamp: "Formatted at 2026-08-06 02:45:38 on dist-test-slave-gmjp"
server_key: "5a44f975c9cbba4779daf39a7a476187"
server_key_iv: "d8ceb6d3124947318f7c847b65e9f57b"
server_key_version: "encryptionkey@0"
I20260806 02:45:38.065312 7344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:38.081756 7344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:38.085106 7344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:38.085354 7344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:38.086035 7344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:38.086982 7344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:38.087059 7344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:38.087136 7344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:38.087188 7344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:38.097208 7344 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.129:40677
I20260806 02:45:38.097232 7539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.129:40677 every 8 connection(s)
I20260806 02:45:38.098358 7344 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
I20260806 02:45:38.107849 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 7344
I20260806 02:45:38.108012 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance
I20260806 02:45:38.108381 1758 external_mini_cluster.cc:1428] Setting key 706ed35fe3e1906d53f0d9b0506d4bad
Aug 06 02:45:38 dist-test-slave-gmjp krb5kdc[7169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984338, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:38.114943 7312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:38.100537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:40639 (local address 127.1.183.190:37317)
0806 02:45:38.100698 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:38.100702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:38.101606 (+ 904us) server_negotiation.cc:407] Connection header received
0806 02:45:38.103168 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:38.103172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:38.103235 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:38.103352 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:38.104727 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:38.105294 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:38.106583 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:38.106824 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:38.106946 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:38.110090 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:38.110095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:38.110098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:38.110129 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:38.112035 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:38.112948 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:38.112951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:38.112952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:38.113037 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:38.113484 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:38.113486 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:38.113487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:38.113680 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:38.113756 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:38.114507 (+ 751us) server_negotiation.cc:299] Negotiation successful
0806 02:45:38.114682 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 02:45:38.115736 7543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:38.100951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:37317 (local address 127.1.183.129:40639)
0806 02:45:38.101433 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:38.101481 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:38.102928 (+ 1447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:38.103552 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:38.103559 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:38.103960 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:38.104565 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:38.104580 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:38.105441 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:38.105451 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:38.106427 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:38.106437 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:38.107002 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:38.107006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:38.107189 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:38.108061 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:38.108087 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:38.109881 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:38.112243 (+ 2362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:38.112250 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:38.112268 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:38.112775 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:38.113141 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:38.113146 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:38.113148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:38.113377 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:38.113798 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:38.113808 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:38.114195 (+ 387us) client_negotiation.cc:769] Sending connection context
0806 02:45:38.114385 (+ 190us) client_negotiation.cc:240] Negotiation successful
0806 02:45:38.114699 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":114,"threads_started":1}
I20260806 02:45:38.117285 7541 heartbeater.cc:347] Connected to a master server at 127.1.183.190:37317
I20260806 02:45:38.117626 7541 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:38.118366 7541 heartbeater.cc:511] Master 127.1.183.190:37317 requested a full tablet report, sending...
I20260806 02:45:38.120518 7254 ts_manager.cc:194] Registered new tserver with Master: e628ed6a84df4cbfa77a81cee9703718 (127.1.183.129:40677)
WARNING: no policy specified for kudu/127.1.183.130@KRBTEST.COM; defaulting to no policy
I20260806 02:45:38.121941 7254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.129@KRBTEST.COM'} at 127.1.183.129:40639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
I20260806 02:45:38.139451 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.130:0
--local_ip_for_outbound_sockets=127.1.183.130
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:37317
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:45879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:38.271675 7550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:38.271976 7550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:38.272023 7550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:38.276350 7550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:38.276451 7550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:38.276553 7550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.130
I20260806 02:45:38.282239 7550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:45879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:37317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.130
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:38.283456 7550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:38.284657 7550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:38.295254 7556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:38.295254 7557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:38.295409 7562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:38.295580 7550 server_base.cc:1034] running on GCE node
I20260806 02:45:38.296103 7550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:38.297000 7550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:38.298205 7550 hybrid_clock.cc:648] HybridClock initialized: now 1785984338298184 us; error 32 us; skew 500 ppm
Aug 06 02:45:38 dist-test-slave-gmjp krb5kdc[7169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984338, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:38.301525 7550 init.cc:377] Logged in from keytab as kudu/127.1.183.130@KRBTEST.COM (short username kudu)
I20260806 02:45:38.302556 7550 webserver.cc:460] Webserver started at http://127.1.183.130:41319/ using document root <none> and password file <none>
I20260806 02:45:38.303174 7550 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:38.303239 7550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:38.303432 7550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:38.305716 7550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-1/data/instance:
uuid: "db344e23d4084671b262125459628540"
format_stamp: "Formatted at 2026-08-06 02:45:38 on dist-test-slave-gmjp"
server_key: "ed40677cdc1ec2e552236125db11fa58"
server_key_iv: "549b9d2396a3d6cbeee0febd6b620c51"
server_key_version: "encryptionkey@0"
I20260806 02:45:38.306300 7550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance:
uuid: "db344e23d4084671b262125459628540"
format_stamp: "Formatted at 2026-08-06 02:45:38 on dist-test-slave-gmjp"
server_key: "ed40677cdc1ec2e552236125db11fa58"
server_key_iv: "549b9d2396a3d6cbeee0febd6b620c51"
server_key_version: "encryptionkey@0"
I20260806 02:45:38.310513 7550 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260806 02:45:38.313365 7571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:38.314716 7550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:38.314895 7550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-1/wal
uuid: "db344e23d4084671b262125459628540"
format_stamp: "Formatted at 2026-08-06 02:45:38 on dist-test-slave-gmjp"
server_key: "ed40677cdc1ec2e552236125db11fa58"
server_key_iv: "549b9d2396a3d6cbeee0febd6b620c51"
server_key_version: "encryptionkey@0"
I20260806 02:45:38.315025 7550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:38.337430 7550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:38.341001 7550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:38.341176 7550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:38.341765 7550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:38.342656 7550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:38.342706 7550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:38.342749 7550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:38.342772 7550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:38.351935 7550 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.130:36065
I20260806 02:45:38.352005 7734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.130:36065 every 8 connection(s)
I20260806 02:45:38.353017 7550 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
I20260806 02:45:38.358011 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 7550
I20260806 02:45:38.358121 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance
I20260806 02:45:38.358429 1758 external_mini_cluster.cc:1428] Setting key c76a4d56f634e8cf78094b0ff13bd072
Aug 06 02:45:38 dist-test-slave-gmjp krb5kdc[7169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984338, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:38.369208 7312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:38.355118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:35143 (local address 127.1.183.190:37317)
0806 02:45:38.355264 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:38.355268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:38.356195 (+ 927us) server_negotiation.cc:407] Connection header received
0806 02:45:38.357739 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:38.357742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:38.357820 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:38.357931 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:38.359392 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:38.360032 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:38.361890 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:38.362127 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:38.362268 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:38.365256 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:38.365260 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:38.365262 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:38.365289 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:38.366808 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:38.367419 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:38.367423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:38.367425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:38.367510 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:38.367925 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:38.367929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:38.367931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:38.368136 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:38.368276 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:38.368859 (+ 583us) server_negotiation.cc:299] Negotiation successful
0806 02:45:38.369034 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.1.183.131@KRBTEST.COM; defaulting to no policy
I20260806 02:45:38.370268 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:38.355452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:37317 (local address 127.1.183.130:35143)
0806 02:45:38.355981 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:38.356030 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:38.357486 (+ 1456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:38.358165 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:38.358175 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:38.358667 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:38.359222 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:38.359244 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:38.360200 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:38.360206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:38.361745 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:38.361761 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:38.362323 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:38.362327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:38.362523 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:38.363197 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:38.363216 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:38.365052 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:38.366978 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:38.366985 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:38.367005 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:38.367298 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:38.367607 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:38.367610 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:38.367612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:38.367780 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:38.368273 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:38.368281 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:38.368600 (+ 319us) client_negotiation.cc:769] Sending connection context
0806 02:45:38.368803 (+ 203us) client_negotiation.cc:240] Negotiation successful
0806 02:45:38.369200 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":126,"threads_started":1}
I20260806 02:45:38.371590 7735 heartbeater.cc:347] Connected to a master server at 127.1.183.190:37317
I20260806 02:45:38.371861 7735 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.131@KRBTEST.COM" created.
I20260806 02:45:38.372551 7735 heartbeater.cc:511] Master 127.1.183.190:37317 requested a full tablet report, sending...
I20260806 02:45:38.373905 7254 ts_manager.cc:194] Registered new tserver with Master: db344e23d4084671b262125459628540 (127.1.183.130:36065)
I20260806 02:45:38.374809 7254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.130@KRBTEST.COM'} at 127.1.183.130:35143
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
I20260806 02:45:38.390892 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.131:0
--local_ip_for_outbound_sockets=127.1.183.131
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:37317
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:45879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:38.514343 7749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:38.514637 7749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:38.514683 7749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:38.519003 7749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:38.519086 7749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:38.519186 7749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.131
I20260806 02:45:38.525372 7749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:45879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:37317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.131
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:38.526589 7749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:38.527746 7749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:38.538376 7763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:38.538331 7760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:38.538466 7759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:38.538940 7749 server_base.cc:1034] running on GCE node
I20260806 02:45:38.539422 7749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:38.540231 7749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:38.541442 7749 hybrid_clock.cc:648] HybridClock initialized: now 1785984338541427 us; error 37 us; skew 500 ppm
Aug 06 02:45:38 dist-test-slave-gmjp krb5kdc[7169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984338, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:38.544512 7749 init.cc:377] Logged in from keytab as kudu/127.1.183.131@KRBTEST.COM (short username kudu)
I20260806 02:45:38.545478 7749 webserver.cc:460] Webserver started at http://127.1.183.131:45085/ using document root <none> and password file <none>
I20260806 02:45:38.546063 7749 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:38.546126 7749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:38.546314 7749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:38.548579 7749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-2/data/instance:
uuid: "ef09816d1014420bad300b09abdcbf67"
format_stamp: "Formatted at 2026-08-06 02:45:38 on dist-test-slave-gmjp"
server_key: "047dd145b6f572e14cde7f98e47054e3"
server_key_iv: "2d103efcef8b8852dcc17fb917221cb4"
server_key_version: "encryptionkey@0"
I20260806 02:45:38.549153 7749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance:
uuid: "ef09816d1014420bad300b09abdcbf67"
format_stamp: "Formatted at 2026-08-06 02:45:38 on dist-test-slave-gmjp"
server_key: "047dd145b6f572e14cde7f98e47054e3"
server_key_iv: "2d103efcef8b8852dcc17fb917221cb4"
server_key_version: "encryptionkey@0"
I20260806 02:45:38.553180 7749 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:45:38.555769 7772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:38.557160 7749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:38.557271 7749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-2/wal
uuid: "ef09816d1014420bad300b09abdcbf67"
format_stamp: "Formatted at 2026-08-06 02:45:38 on dist-test-slave-gmjp"
server_key: "047dd145b6f572e14cde7f98e47054e3"
server_key_iv: "2d103efcef8b8852dcc17fb917221cb4"
server_key_version: "encryptionkey@0"
I20260806 02:45:38.557366 7749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:38.573328 7749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:38.576695 7749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:38.576871 7749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:38.577481 7749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:38.578336 7749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:38.578394 7749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:38.578438 7749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:38.578475 7749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:38.588342 7749 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.131:41095
I20260806 02:45:38.588373 7941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.131:41095 every 8 connection(s)
I20260806 02:45:38.589378 7749 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
I20260806 02:45:38.599535 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 7749
Aug 06 02:45:38 dist-test-slave-gmjp krb5kdc[7169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984338, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:38.599700 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance
I20260806 02:45:38.600063 1758 external_mini_cluster.cc:1428] Setting key 2e57fb6f9cdf58cb66f455b2ce5a7ec9
I20260806 02:45:38.604699 7312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:38.591484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:41599 (local address 127.1.183.190:37317)
0806 02:45:38.591634 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:38.591638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:38.592678 (+ 1040us) server_negotiation.cc:407] Connection header received
0806 02:45:38.594055 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:38.594058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:38.594112 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:38.594192 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:38.595408 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:38.595936 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:38.597103 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:38.597318 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:38.597407 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:38.600449 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:38.600453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:38.600455 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:38.600509 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:38.602370 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:38.603048 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:38.603051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:38.603053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:38.603134 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:38.603458 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:38.603461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:38.603462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:38.603647 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:38.603721 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:38.604450 (+ 729us) server_negotiation.cc:299] Negotiation successful
0806 02:45:38.604568 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:45:38.605566 7947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:38.591837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:37317 (local address 127.1.183.131:41599)
0806 02:45:38.592500 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:38.592548 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:38.593849 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:38.594365 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:38.594372 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:38.594773 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:38.595264 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:38.595277 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:38.596049 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:38.596051 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:38.596990 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:38.596999 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:38.597481 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:38.597484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:38.597627 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:38.598330 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:38.598347 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:38.600245 (+ 1898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:38.602518 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:38.602529 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:38.602549 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:38.602937 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:38.603215 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:38.603217 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:38.603218 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:38.603362 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:38.603735 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:38.603740 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:38.604103 (+ 363us) client_negotiation.cc:769] Sending connection context
0806 02:45:38.604343 (+ 240us) client_negotiation.cc:240] Negotiation successful
0806 02:45:38.604649 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":122,"threads_started":1}
I20260806 02:45:38.606750 7942 heartbeater.cc:347] Connected to a master server at 127.1.183.190:37317
I20260806 02:45:38.607030 7942 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:38.607586 7942 heartbeater.cc:511] Master 127.1.183.190:37317 requested a full tablet report, sending...
I20260806 02:45:38.608755 7254 ts_manager.cc:194] Registered new tserver with Master: ef09816d1014420bad300b09abdcbf67 (127.1.183.131:41095)
I20260806 02:45:38.609337 7254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.131@KRBTEST.COM'} at 127.1.183.131:41599
I20260806 02:45:38.614743 1758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:45:38 dist-test-slave-gmjp krb5kdc[7169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:45:38 dist-test-slave-gmjp krb5kdc[7169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.129@KRBTEST.COM
I20260806 02:45:38.644337 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:38.633667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49478 (local address 127.1.183.129:40677)
0806 02:45:38.633972 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:38.633982 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:38.634012 (+ 30us) server_negotiation.cc:407] Connection header received
0806 02:45:38.634133 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:38.634139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:38.634343 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:38.634518 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:38.635292 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:38.636069 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:38.636990 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:38.637411 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:38.637535 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:38.640303 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:38.640316 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:38.640333 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:38.640365 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:38.642231 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:38.642708 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:38.642713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:38.642714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:38.642803 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:38.643144 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:38.643146 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:38.643147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:38.643414 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:38.643588 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:38.643893 (+ 305us) server_negotiation.cc:299] Negotiation successful
0806 02:45:38.644087 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":72,"threads_started":1}
Aug 06 02:45:38 dist-test-slave-gmjp krb5kdc[7169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:38.656560 7312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:38.647461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52222 (local address 127.1.183.190:37317)
0806 02:45:38.647636 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:38.647645 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:38.647664 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:45:38.647853 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:38.647859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:38.647915 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:38.648011 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:38.648849 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:38.649417 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:38.650270 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:38.650514 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:38.650642 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:38.653356 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:38.653360 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:38.653362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:38.653394 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:38.654728 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:38.655198 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:38.655207 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:38.655208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:38.655287 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:38.655582 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:38.655586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:38.655588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:38.655840 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:38.655988 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:38.656247 (+ 259us) server_negotiation.cc:299] Negotiation successful
0806 02:45:38.656385 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260806 02:45:38.657039 7254 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52222
I20260806 02:45:38.657765 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 7344
I20260806 02:45:38.675164 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 7550
I20260806 02:45:38.688102 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 7749
I20260806 02:45:38.701210 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 7197
2026-08-06T02:45:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2561 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:45:38 dist-test-slave-gmjp krb5kdc[7961](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:45:38 dist-test-slave-gmjp krb5kdc[7961](info): set up 2 sockets
Aug 06 02:45:38 dist-test-slave-gmjp krb5kdc[7961](info): commencing operation
krb5kdc: starting...
W20260806 02:45:39.943732 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.210s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:45:40 dist-test-slave-gmjp krb5kdc[7961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:45:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
I20260806 02:45:40.079742 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:33007
--webserver_interface=127.1.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:33007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:40.200497 7978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:40.200820 7978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:40.200917 7978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:40.205351 7978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:40.205430 7978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:40.205451 7978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:40.205498 7978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:40.205531 7978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:40.211154 7978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:33007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:33007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:40.212483 7978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:40.213644 7978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:40.222486 7984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:40.222525 7985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:40.222453 7987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:40.222891 7978 server_base.cc:1034] running on GCE node
I20260806 02:45:40.223596 7978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:40.224881 7978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:40.226053 7978 hybrid_clock.cc:648] HybridClock initialized: now 1785984340226034 us; error 33 us; skew 500 ppm
Aug 06 02:45:40 dist-test-slave-gmjp krb5kdc[7961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984340, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:40.233027 7978 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:45:40.233945 7978 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:40.234035 7978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:40.234287 7978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:40.236644 7978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "032c8a141aca4a05a50314d24b6c34ec"
format_stamp: "Formatted at 2026-08-06 02:45:40 on dist-test-slave-gmjp"
server_key: "f3492e73c1b92cfc00ce00c27a8c857f"
server_key_iv: "fa9617124ca6289e91b3896740c5f0c1"
server_key_version: "encryptionkey@0"
I20260806 02:45:40.237260 7978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "032c8a141aca4a05a50314d24b6c34ec"
format_stamp: "Formatted at 2026-08-06 02:45:40 on dist-test-slave-gmjp"
server_key: "f3492e73c1b92cfc00ce00c27a8c857f"
server_key_iv: "fa9617124ca6289e91b3896740c5f0c1"
server_key_version: "encryptionkey@0"
I20260806 02:45:40.241395 7978 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:45:40.244040 7996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:40.245422 7978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:40.245585 7978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "032c8a141aca4a05a50314d24b6c34ec"
format_stamp: "Formatted at 2026-08-06 02:45:40 on dist-test-slave-gmjp"
server_key: "f3492e73c1b92cfc00ce00c27a8c857f"
server_key_iv: "fa9617124ca6289e91b3896740c5f0c1"
server_key_version: "encryptionkey@0"
I20260806 02:45:40.245714 7978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:40.262236 7978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:40.266003 7978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:40.266237 7978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:40.273622 7978 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:33007
I20260806 02:45:40.273929 8074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:33007 every 8 connection(s)
I20260806 02:45:40.274384 7978 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:40.277432 8077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:40.277969 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 7978
I20260806 02:45:40.278152 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:45:40.278498 1758 external_mini_cluster.cc:1428] Setting key d9630459eb9306d62ae42ae850a6af55
I20260806 02:45:40.284834 8077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 032c8a141aca4a05a50314d24b6c34ec: Bootstrap starting.
I20260806 02:45:40.287411 8077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 032c8a141aca4a05a50314d24b6c34ec: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:40.288291 8077 log.cc:824] T 00000000000000000000000000000000 P 032c8a141aca4a05a50314d24b6c34ec: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:40.290540 8077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 032c8a141aca4a05a50314d24b6c34ec: No bootstrap required, opened a new log
Aug 06 02:45:40 dist-test-slave-gmjp krb5kdc[7961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:40.294174 8077 raft_consensus.cc:348] T 00000000000000000000000000000000 P 032c8a141aca4a05a50314d24b6c34ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032c8a141aca4a05a50314d24b6c34ec" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33007 } }
I20260806 02:45:40.294427 8077 raft_consensus.cc:374] T 00000000000000000000000000000000 P 032c8a141aca4a05a50314d24b6c34ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:40.294507 8077 raft_consensus.cc:729] T 00000000000000000000000000000000 P 032c8a141aca4a05a50314d24b6c34ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032c8a141aca4a05a50314d24b6c34ec, State: Initialized, Role: FOLLOWER
I20260806 02:45:40.295084 8077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 032c8a141aca4a05a50314d24b6c34ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032c8a141aca4a05a50314d24b6c34ec" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33007 } }
I20260806 02:45:40.295274 8077 raft_consensus.cc:388] T 00000000000000000000000000000000 P 032c8a141aca4a05a50314d24b6c34ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:40.295357 8077 raft_consensus.cc:482] T 00000000000000000000000000000000 P 032c8a141aca4a05a50314d24b6c34ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:40.295485 8077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 032c8a141aca4a05a50314d24b6c34ec [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:40.296808 8077 raft_consensus.cc:504] T 00000000000000000000000000000000 P 032c8a141aca4a05a50314d24b6c34ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032c8a141aca4a05a50314d24b6c34ec" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33007 } }
I20260806 02:45:40.297223 8077 leader_election.cc:302] T 00000000000000000000000000000000 P 032c8a141aca4a05a50314d24b6c34ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 032c8a141aca4a05a50314d24b6c34ec; no voters:
I20260806 02:45:40.297605 8077 leader_election.cc:288] T 00000000000000000000000000000000 P 032c8a141aca4a05a50314d24b6c34ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:40.297732 8085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 032c8a141aca4a05a50314d24b6c34ec [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:40.297951 8085 raft_consensus.cc:686] T 00000000000000000000000000000000 P 032c8a141aca4a05a50314d24b6c34ec [term 1 LEADER]: Becoming Leader. State: Replica: 032c8a141aca4a05a50314d24b6c34ec, State: Running, Role: LEADER
I20260806 02:45:40.298744 8081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:40.280003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39940 (local address 127.1.183.190:33007)
0806 02:45:40.280513 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:40.280530 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:40.280567 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:45:40.281277 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:40.281296 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:40.281647 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:40.282023 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:40.282883 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:40.283902 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:40.284896 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:40.285441 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:40.285635 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:40.292215 (+ 6580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:40.292228 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:40.292245 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:40.292277 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:40.294872 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:40.295483 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:40.295490 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:40.295497 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:40.295683 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:40.296015 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:40.296018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:40.296019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:40.296535 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:40.296688 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:40.297124 (+ 436us) server_negotiation.cc:299] Negotiation successful
0806 02:45:40.297479 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":359,"thread_start_us":190,"threads_started":1}
I20260806 02:45:40.299084 8085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 032c8a141aca4a05a50314d24b6c34ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032c8a141aca4a05a50314d24b6c34ec" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33007 } }
I20260806 02:45:40.299082 8077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 032c8a141aca4a05a50314d24b6c34ec [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:40.301190 8086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032c8a141aca4a05a50314d24b6c34ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "032c8a141aca4a05a50314d24b6c34ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032c8a141aca4a05a50314d24b6c34ec" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33007 } } }
I20260806 02:45:40.301190 8087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032c8a141aca4a05a50314d24b6c34ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 032c8a141aca4a05a50314d24b6c34ec. Latest consensus state: current_term: 1 leader_uuid: "032c8a141aca4a05a50314d24b6c34ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032c8a141aca4a05a50314d24b6c34ec" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33007 } } }
I20260806 02:45:40.301390 8086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032c8a141aca4a05a50314d24b6c34ec [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:40.301448 8087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032c8a141aca4a05a50314d24b6c34ec [sys.catalog]: This master's current role is: LEADER
W20260806 02:45:40.304143 8109 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 032c8a141aca4a05a50314d24b6c34ec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:45:40.304391 8109 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:45:40.304543 8112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:40.305492 8112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:40.311378 8112 catalog_manager.cc:1347] Generated new cluster ID: 6637c06fbc144073a68fcec2e3088790
I20260806 02:45:40.311477 8112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:40.321977 8112 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:40.323375 8112 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:40.335991 8112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 032c8a141aca4a05a50314d24b6c34ec: Generated new TSK 0
I20260806 02:45:40.337035 8112 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
I20260806 02:45:40.379123 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.129:0
--local_ip_for_outbound_sockets=127.1.183.129
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:33007
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:40.517133 8120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:40.517606 8120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:40.517709 8120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:40.524871 8120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:40.524960 8120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:40.525070 8120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.129
I20260806 02:45:40.530972 8120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:33007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.129
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:40.532615 8120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:40.533847 8120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:40.544284 8128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:40.544284 8127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:40.544317 8130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:40.544490 8120 server_base.cc:1034] running on GCE node
I20260806 02:45:40.545574 8120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:40.546337 8120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:40.547539 8120 hybrid_clock.cc:648] HybridClock initialized: now 1785984340547507 us; error 54 us; skew 500 ppm
Aug 06 02:45:40 dist-test-slave-gmjp krb5kdc[7961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984340, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:40.551019 8120 init.cc:377] Logged in from keytab as kudu/127.1.183.129@KRBTEST.COM (short username kudu)
I20260806 02:45:40.551822 8120 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:40.551900 8120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:40.552081 8120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:40.554389 8120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-0/data/instance:
uuid: "ed9ec92f5ff745ba9705e2a711320b7b"
format_stamp: "Formatted at 2026-08-06 02:45:40 on dist-test-slave-gmjp"
server_key: "0dca951c1ad499986ddfb9ba2d1ddf23"
server_key_iv: "0c3bd84e98afe64a1c3565472c15e542"
server_key_version: "encryptionkey@0"
I20260806 02:45:40.555014 8120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance:
uuid: "ed9ec92f5ff745ba9705e2a711320b7b"
format_stamp: "Formatted at 2026-08-06 02:45:40 on dist-test-slave-gmjp"
server_key: "0dca951c1ad499986ddfb9ba2d1ddf23"
server_key_iv: "0c3bd84e98afe64a1c3565472c15e542"
server_key_version: "encryptionkey@0"
I20260806 02:45:40.559230 8120 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:45:40.562077 8142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:40.563418 8120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:45:40.563562 8120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-0/wal
uuid: "ed9ec92f5ff745ba9705e2a711320b7b"
format_stamp: "Formatted at 2026-08-06 02:45:40 on dist-test-slave-gmjp"
server_key: "0dca951c1ad499986ddfb9ba2d1ddf23"
server_key_iv: "0c3bd84e98afe64a1c3565472c15e542"
server_key_version: "encryptionkey@0"
I20260806 02:45:40.563687 8120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:40.594523 8120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:40.598119 8120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:40.598325 8120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:40.599020 8120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:40.600077 8120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:40.600127 8120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:40.600252 8120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:40.600291 8120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:40.610015 8120 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.129:34869
I20260806 02:45:40.610035 8311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.129:34869 every 8 connection(s)
I20260806 02:45:40.610692 8120 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
I20260806 02:45:40.611929 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 8120
I20260806 02:45:40.612046 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance
I20260806 02:45:40.612411 1758 external_mini_cluster.cc:1428] Setting key 27e0bf3630feb3b247f593900737f509
Aug 06 02:45:40 dist-test-slave-gmjp krb5kdc[7961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984340, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.130@KRBTEST.COM" created.
I20260806 02:45:40.627208 8081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:40.612720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:45913 (local address 127.1.183.190:33007)
0806 02:45:40.612890 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:40.612895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:40.613929 (+ 1034us) server_negotiation.cc:407] Connection header received
0806 02:45:40.615378 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:40.615384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:40.615452 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:40.615548 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:40.616902 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:40.617602 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:40.618846 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:40.619294 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:40.619435 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:40.622791 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:40.622795 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:40.622797 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:40.622826 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:40.624598 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:40.625334 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:40.625336 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:40.625338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:40.625424 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:40.625790 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:40.625793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:40.625794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:40.626012 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:40.626203 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:40.626850 (+ 647us) server_negotiation.cc:299] Negotiation successful
0806 02:45:40.627042 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 02:45:40.628427 8314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:40.613236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:33007 (local address 127.1.183.129:45913)
0806 02:45:40.613755 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:40.613813 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:40.615153 (+ 1340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:40.615730 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:40.615737 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:40.616205 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:40.616737 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:40.616752 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:40.617751 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:40.617754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:40.618686 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:40.618696 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:40.619478 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:40.619481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:40.619687 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:40.620475 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:40.620502 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:40.622538 (+ 2036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:40.624821 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:40.624827 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:40.624844 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:40.625226 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:40.625515 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:40.625517 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:40.625519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:40.625670 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:40.626210 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:40.626216 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:40.626523 (+ 307us) client_negotiation.cc:769] Sending connection context
0806 02:45:40.626712 (+ 189us) client_negotiation.cc:240] Negotiation successful
0806 02:45:40.627196 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":142,"threads_started":1}
I20260806 02:45:40.630144 8312 heartbeater.cc:347] Connected to a master server at 127.1.183.190:33007
I20260806 02:45:40.630477 8312 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:40.631145 8312 heartbeater.cc:511] Master 127.1.183.190:33007 requested a full tablet report, sending...
I20260806 02:45:40.633176 8023 ts_manager.cc:194] Registered new tserver with Master: ed9ec92f5ff745ba9705e2a711320b7b (127.1.183.129:34869)
I20260806 02:45:40.634704 8023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.129@KRBTEST.COM'} at 127.1.183.129:45913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
I20260806 02:45:40.648015 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.130:0
--local_ip_for_outbound_sockets=127.1.183.130
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:33007
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:40.767244 8319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:40.767550 8319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:40.767627 8319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:40.772073 8319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:40.772199 8319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:40.772330 8319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.130
I20260806 02:45:40.777959 8319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:33007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.130
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:40.779232 8319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:40.780402 8319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:40.790833 8333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:40.790834 8330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:40.790861 8328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:40.791208 8319 server_base.cc:1034] running on GCE node
I20260806 02:45:40.791661 8319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:40.792503 8319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:40.793710 8319 hybrid_clock.cc:648] HybridClock initialized: now 1785984340793687 us; error 37 us; skew 500 ppm
Aug 06 02:45:40 dist-test-slave-gmjp krb5kdc[7961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984340, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:40.796823 8319 init.cc:377] Logged in from keytab as kudu/127.1.183.130@KRBTEST.COM (short username kudu)
I20260806 02:45:40.797823 8319 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:40.797938 8319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:40.798194 8319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:40.800657 8319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-1/data/instance:
uuid: "c9e74808dc05428498ef00d1f3d45b09"
format_stamp: "Formatted at 2026-08-06 02:45:40 on dist-test-slave-gmjp"
server_key: "6598eaeb0d007c4feda73b9836750d07"
server_key_iv: "3ff15ba53601231c3562ed5e07d004cf"
server_key_version: "encryptionkey@0"
I20260806 02:45:40.801425 8319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance:
uuid: "c9e74808dc05428498ef00d1f3d45b09"
format_stamp: "Formatted at 2026-08-06 02:45:40 on dist-test-slave-gmjp"
server_key: "6598eaeb0d007c4feda73b9836750d07"
server_key_iv: "3ff15ba53601231c3562ed5e07d004cf"
server_key_version: "encryptionkey@0"
I20260806 02:45:40.805652 8319 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:45:40.808506 8341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:40.809859 8319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:45:40.810040 8319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-1/wal
uuid: "c9e74808dc05428498ef00d1f3d45b09"
format_stamp: "Formatted at 2026-08-06 02:45:40 on dist-test-slave-gmjp"
server_key: "6598eaeb0d007c4feda73b9836750d07"
server_key_iv: "3ff15ba53601231c3562ed5e07d004cf"
server_key_version: "encryptionkey@0"
I20260806 02:45:40.810177 8319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:40.855067 8319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:40.858903 8319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:40.859148 8319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:40.859865 8319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:40.860859 8319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:40.860956 8319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:40.861056 8319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:40.861109 8319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:40.871100 8319 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.130:33761
I20260806 02:45:40.871120 8500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.130:33761 every 8 connection(s)
I20260806 02:45:40.871773 8319 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
I20260806 02:45:40.877341 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 8319
I20260806 02:45:40.877503 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance
I20260806 02:45:40.877836 1758 external_mini_cluster.cc:1428] Setting key 4fb2c0c1272a5665c78d11b21c5f272d
Aug 06 02:45:40 dist-test-slave-gmjp krb5kdc[7961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984340, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:40.887727 8081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:40.873784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:38735 (local address 127.1.183.190:33007)
0806 02:45:40.873945 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:40.873949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:40.874869 (+ 920us) server_negotiation.cc:407] Connection header received
0806 02:45:40.876316 (+ 1447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:40.876318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:40.876369 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:40.876448 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:40.877809 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:40.878530 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:40.880130 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:40.880426 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:40.880510 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:40.883385 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:40.883390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:40.883394 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:40.883430 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:40.885299 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:40.885938 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:40.885945 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:40.885948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:40.886054 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:40.886415 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:40.886422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:40.886426 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:40.886717 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:40.886842 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:40.887377 (+ 535us) server_negotiation.cc:299] Negotiation successful
0806 02:45:40.887521 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:45:40.888635 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:40.874183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:33007 (local address 127.1.183.130:38735)
0806 02:45:40.874685 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:40.874731 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:40.876039 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:40.876631 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:40.876638 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:40.877040 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:40.877639 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:40.877657 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:40.878692 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:40.878695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:40.879980 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:40.879995 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:40.880596 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:40.880603 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:40.880852 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:40.881569 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:40.881598 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:40.883193 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:40.885455 (+ 2262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:40.885462 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:40.885478 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:40.885805 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:40.886165 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:40.886167 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:40.886169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:40.886315 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:40.886846 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:40.886852 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:40.887112 (+ 260us) client_negotiation.cc:769] Sending connection context
0806 02:45:40.887280 (+ 168us) client_negotiation.cc:240] Negotiation successful
0806 02:45:40.887561 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":127,"threads_started":1}
WARNING: no policy specified for kudu/127.1.183.131@KRBTEST.COM; defaulting to no policy
I20260806 02:45:40.890038 8501 heartbeater.cc:347] Connected to a master server at 127.1.183.190:33007
I20260806 02:45:40.890333 8501 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:40.890909 8501 heartbeater.cc:511] Master 127.1.183.190:33007 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.131@KRBTEST.COM" created.
I20260806 02:45:40.892310 8023 ts_manager.cc:194] Registered new tserver with Master: c9e74808dc05428498ef00d1f3d45b09 (127.1.183.130:33761)
I20260806 02:45:40.893105 8023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.130@KRBTEST.COM'} at 127.1.183.130:38735
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
I20260806 02:45:40.912097 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.131:0
--local_ip_for_outbound_sockets=127.1.183.131
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:33007
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:41.033871 8513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:41.034178 8513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:41.034265 8513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:41.038470 8513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:41.038555 8513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:41.038620 8513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.131
I20260806 02:45:41.044615 8513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:33007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.131
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:41.045921 8513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:41.047065 8513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:41.057636 8527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:41.057624 8530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:41.057624 8533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:41.057729 8513 server_base.cc:1034] running on GCE node
I20260806 02:45:41.058490 8513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:41.059209 8513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:41.060410 8513 hybrid_clock.cc:648] HybridClock initialized: now 1785984341060402 us; error 43 us; skew 500 ppm
Aug 06 02:45:41 dist-test-slave-gmjp krb5kdc[7961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984341, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:41.063439 8513 init.cc:377] Logged in from keytab as kudu/127.1.183.131@KRBTEST.COM (short username kudu)
I20260806 02:45:41.064328 8513 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:41.064435 8513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:41.064693 8513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:41.067077 8513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-2/data/instance:
uuid: "6b9fc178691142388ea108e9e5f8a8c0"
format_stamp: "Formatted at 2026-08-06 02:45:41 on dist-test-slave-gmjp"
server_key: "7e068edc1d62270da774062bc9e78ac9"
server_key_iv: "a9d847aa775a2f9bd0047c615337ae9e"
server_key_version: "encryptionkey@0"
I20260806 02:45:41.067778 8513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance:
uuid: "6b9fc178691142388ea108e9e5f8a8c0"
format_stamp: "Formatted at 2026-08-06 02:45:41 on dist-test-slave-gmjp"
server_key: "7e068edc1d62270da774062bc9e78ac9"
server_key_iv: "a9d847aa775a2f9bd0047c615337ae9e"
server_key_version: "encryptionkey@0"
I20260806 02:45:41.072206 8513 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 02:45:41.074999 8542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:41.076570 8513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:41.076743 8513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-2/wal
uuid: "6b9fc178691142388ea108e9e5f8a8c0"
format_stamp: "Formatted at 2026-08-06 02:45:41 on dist-test-slave-gmjp"
server_key: "7e068edc1d62270da774062bc9e78ac9"
server_key_iv: "a9d847aa775a2f9bd0047c615337ae9e"
server_key_version: "encryptionkey@0"
I20260806 02:45:41.076920 8513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:41.095391 8513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:41.099179 8513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:41.099471 8513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:41.100325 8513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:41.101441 8513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:41.101511 8513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:41.101576 8513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:41.101629 8513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:41.111922 8513 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.131:35687
I20260806 02:45:41.111951 8708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.131:35687 every 8 connection(s)
I20260806 02:45:41.112675 8513 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
I20260806 02:45:41.121220 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 8513
I20260806 02:45:41.121378 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance
I20260806 02:45:41.121739 1758 external_mini_cluster.cc:1428] Setting key 542ca4f637480d278d5e2c01e3cda0e3
Aug 06 02:45:41 dist-test-slave-gmjp krb5kdc[7961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984341, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:41.128412 8081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:41.114861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:50751 (local address 127.1.183.190:33007)
0806 02:45:41.115068 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:41.115072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:41.115986 (+ 914us) server_negotiation.cc:407] Connection header received
0806 02:45:41.117470 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:41.117473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:41.117539 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:41.117625 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:41.119077 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:41.119679 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:41.121154 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:41.121523 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:41.121653 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:41.124871 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:41.124874 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:41.124876 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:41.124904 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:41.126289 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:41.126835 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:41.126838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:41.126839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:41.126925 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:41.127259 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:41.127262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:41.127263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:41.127452 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:41.127526 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:41.128069 (+ 543us) server_negotiation.cc:299] Negotiation successful
0806 02:45:41.128276 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260806 02:45:41.129391 8711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:41.115259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:33007 (local address 127.1.183.131:50751)
0806 02:45:41.115805 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:41.115855 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:41.117201 (+ 1346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:41.117850 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:41.117859 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:41.118305 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:41.118865 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:41.118881 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:41.119831 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:41.119835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:41.120991 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:41.121003 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:41.121702 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:41.121705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:41.121917 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:41.122689 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:41.122714 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:41.124661 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:41.126407 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:41.126412 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:41.126433 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:41.126719 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:41.127013 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:41.127015 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:41.127016 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:41.127167 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:41.127538 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:41.127544 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:41.127802 (+ 258us) client_negotiation.cc:769] Sending connection context
0806 02:45:41.127983 (+ 181us) client_negotiation.cc:240] Negotiation successful
0806 02:45:41.128371 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":132,"threads_started":1}
I20260806 02:45:41.130681 8709 heartbeater.cc:347] Connected to a master server at 127.1.183.190:33007
I20260806 02:45:41.130913 8709 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:41.131412 8709 heartbeater.cc:511] Master 127.1.183.190:33007 requested a full tablet report, sending...
I20260806 02:45:41.132557 8023 ts_manager.cc:194] Registered new tserver with Master: 6b9fc178691142388ea108e9e5f8a8c0 (127.1.183.131:35687)
I20260806 02:45:41.133247 8023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.131@KRBTEST.COM'} at 127.1.183.131:50751
I20260806 02:45:41.136077 1758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:45:41.146281 8081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:41.138188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39944 (local address 127.1.183.190:33007)
0806 02:45:41.138347 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:41.138351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:41.138575 (+ 224us) server_negotiation.cc:407] Connection header received
0806 02:45:41.138677 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:41.138679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:41.138719 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:41.138818 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:41.139546 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:41.140109 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:41.141105 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:41.141327 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:41.141413 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:41.142884 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:41.142887 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:41.142889 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:41.142915 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:41.144434 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:41.144873 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:41.144876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:41.144877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:41.144969 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:41.145348 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:41.145351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:41.145352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:41.145615 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:41.145773 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:41.146000 (+ 227us) server_negotiation.cc:299] Negotiation successful
0806 02:45:41.146125 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 02:45:41.149802 8023 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:45:41.153110 8023 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:45:41.169037 8731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:41.163596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40344 (local address 127.1.183.130:33761)
0806 02:45:41.164073 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:41.164085 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:41.164108 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:45:41.164323 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:41.164329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:41.164531 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:41.164688 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:41.164904 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:41.165986 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:41.167619 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:41.168269 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:41.168423 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:41.168597 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:41.168744 (+ 147us) server_negotiation.cc:299] Negotiation successful
0806 02:45:41.168854 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":160,"threads_started":1}
I20260806 02:45:41.169296 8727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:41.162730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34212 (local address 127.1.183.129:34869)
0806 02:45:41.163209 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:41.163219 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:41.163480 (+ 261us) server_negotiation.cc:407] Connection header received
0806 02:45:41.163570 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:41.163576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:41.163815 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:41.163987 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:41.164514 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:41.165725 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:41.167604 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:41.168452 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:41.168548 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:41.168669 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:41.169033 (+ 364us) server_negotiation.cc:299] Negotiation successful
0806 02:45:41.169134 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":130,"threads_started":1}
I20260806 02:45:41.169032 8732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:41.163421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:33761 (local address 127.0.0.1:40344)
0806 02:45:41.164019 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:41.164034 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:41.164124 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:41.164678 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:41.164680 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:41.164700 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:41.164803 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:41.164809 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:41.166141 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:41.166144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:41.167461 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:41.167472 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:41.168423 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:41.168428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:41.168546 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:41.168586 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 02:45:41.168712 (+ 126us) client_negotiation.cc:240] Negotiation successful
0806 02:45:41.168860 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":56,"thread_start_us":104,"threads_started":1}
I20260806 02:45:41.169637 8730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:41.163791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:35687 (local address 127.0.0.1:40296)
0806 02:45:41.164295 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:41.164309 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:41.164400 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:41.165001 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:41.165004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:41.165021 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:41.165122 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:41.165128 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:41.166256 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:41.166259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:41.167267 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:41.167277 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:41.168670 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:41.168675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:41.168768 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:41.168792 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 02:45:41.168865 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 02:45:41.168950 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":1,"thread_start_us":82,"threads_started":1}
I20260806 02:45:41.169807 8733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:41.164101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40296 (local address 127.1.183.131:35687)
0806 02:45:41.164565 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:41.164574 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:41.164592 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:45:41.164678 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:41.164683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:41.164889 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:41.165024 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:41.165233 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:41.166110 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:41.167395 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:41.168335 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:41.168446 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:41.168604 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:41.169523 (+ 919us) server_negotiation.cc:299] Negotiation successful
0806 02:45:41.169638 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":141,"threads_started":1}
I20260806 02:45:41.169842 8726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:41.162563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:34869 (local address 127.0.0.1:34212)
0806 02:45:41.162928 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:41.162981 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:41.163112 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:41.164204 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:41.164212 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:41.164236 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:41.164377 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:41.164384 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:41.165871 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:41.165874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:41.167461 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:41.167472 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:41.168719 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:41.168722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:41.168810 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:41.168844 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 02:45:41.168908 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 02:45:41.168963 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":64,"thread_start_us":115,"threads_started":1}
I20260806 02:45:41.172640 8614 tablet_service.cc:1467] Processing CreateTablet for tablet d4b1696a4b98403686a94e1e3abd7d3a (DEFAULT_TABLE table=test-table [id=b8f1af02a5c04f619d1cfc0dc67d0999]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:41.172720 8410 tablet_service.cc:1467] Processing CreateTablet for tablet d4b1696a4b98403686a94e1e3abd7d3a (DEFAULT_TABLE table=test-table [id=b8f1af02a5c04f619d1cfc0dc67d0999]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:41.173146 8215 tablet_service.cc:1467] Processing CreateTablet for tablet d4b1696a4b98403686a94e1e3abd7d3a (DEFAULT_TABLE table=test-table [id=b8f1af02a5c04f619d1cfc0dc67d0999]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:41.173822 8614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b1696a4b98403686a94e1e3abd7d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:41.173861 8410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b1696a4b98403686a94e1e3abd7d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:41.174093 8215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b1696a4b98403686a94e1e3abd7d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:41.180490 8734 tablet_bootstrap.cc:492] T d4b1696a4b98403686a94e1e3abd7d3a P 6b9fc178691142388ea108e9e5f8a8c0: Bootstrap starting.
I20260806 02:45:41.182271 8736 tablet_bootstrap.cc:492] T d4b1696a4b98403686a94e1e3abd7d3a P ed9ec92f5ff745ba9705e2a711320b7b: Bootstrap starting.
I20260806 02:45:41.182510 8737 tablet_bootstrap.cc:492] T d4b1696a4b98403686a94e1e3abd7d3a P c9e74808dc05428498ef00d1f3d45b09: Bootstrap starting.
I20260806 02:45:41.183445 8734 tablet_bootstrap.cc:654] T d4b1696a4b98403686a94e1e3abd7d3a P 6b9fc178691142388ea108e9e5f8a8c0: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:41.185070 8736 tablet_bootstrap.cc:654] T d4b1696a4b98403686a94e1e3abd7d3a P ed9ec92f5ff745ba9705e2a711320b7b: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:41.185882 8737 tablet_bootstrap.cc:654] T d4b1696a4b98403686a94e1e3abd7d3a P c9e74808dc05428498ef00d1f3d45b09: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:41.186090 8736 log.cc:824] T d4b1696a4b98403686a94e1e3abd7d3a P ed9ec92f5ff745ba9705e2a711320b7b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:41.186329 8734 log.cc:824] T d4b1696a4b98403686a94e1e3abd7d3a P 6b9fc178691142388ea108e9e5f8a8c0: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:41.187243 8737 log.cc:824] T d4b1696a4b98403686a94e1e3abd7d3a P c9e74808dc05428498ef00d1f3d45b09: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:41.189502 8734 tablet_bootstrap.cc:492] T d4b1696a4b98403686a94e1e3abd7d3a P 6b9fc178691142388ea108e9e5f8a8c0: No bootstrap required, opened a new log
I20260806 02:45:41.189535 8736 tablet_bootstrap.cc:492] T d4b1696a4b98403686a94e1e3abd7d3a P ed9ec92f5ff745ba9705e2a711320b7b: No bootstrap required, opened a new log
I20260806 02:45:41.189786 8734 ts_tablet_manager.cc:1397] T d4b1696a4b98403686a94e1e3abd7d3a P 6b9fc178691142388ea108e9e5f8a8c0: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260806 02:45:41.189793 8736 ts_tablet_manager.cc:1397] T d4b1696a4b98403686a94e1e3abd7d3a P ed9ec92f5ff745ba9705e2a711320b7b: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260806 02:45:41.190496 8737 tablet_bootstrap.cc:492] T d4b1696a4b98403686a94e1e3abd7d3a P c9e74808dc05428498ef00d1f3d45b09: No bootstrap required, opened a new log
I20260806 02:45:41.190832 8737 ts_tablet_manager.cc:1397] T d4b1696a4b98403686a94e1e3abd7d3a P c9e74808dc05428498ef00d1f3d45b09: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260806 02:45:41.194087 8736 raft_consensus.cc:348] T d4b1696a4b98403686a94e1e3abd7d3a P ed9ec92f5ff745ba9705e2a711320b7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9ec92f5ff745ba9705e2a711320b7b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 34869 } } peers { permanent_uuid: "c9e74808dc05428498ef00d1f3d45b09" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33761 } } peers { permanent_uuid: "6b9fc178691142388ea108e9e5f8a8c0" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 35687 } }
I20260806 02:45:41.194388 8736 raft_consensus.cc:374] T d4b1696a4b98403686a94e1e3abd7d3a P ed9ec92f5ff745ba9705e2a711320b7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:41.194444 8736 raft_consensus.cc:729] T d4b1696a4b98403686a94e1e3abd7d3a P ed9ec92f5ff745ba9705e2a711320b7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed9ec92f5ff745ba9705e2a711320b7b, State: Initialized, Role: FOLLOWER
I20260806 02:45:41.194945 8736 consensus_queue.cc:260] T d4b1696a4b98403686a94e1e3abd7d3a P ed9ec92f5ff745ba9705e2a711320b7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9ec92f5ff745ba9705e2a711320b7b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 34869 } } peers { permanent_uuid: "c9e74808dc05428498ef00d1f3d45b09" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33761 } } peers { permanent_uuid: "6b9fc178691142388ea108e9e5f8a8c0" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 35687 } }
I20260806 02:45:41.195067 8734 raft_consensus.cc:348] T d4b1696a4b98403686a94e1e3abd7d3a P 6b9fc178691142388ea108e9e5f8a8c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9ec92f5ff745ba9705e2a711320b7b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 34869 } } peers { permanent_uuid: "c9e74808dc05428498ef00d1f3d45b09" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33761 } } peers { permanent_uuid: "6b9fc178691142388ea108e9e5f8a8c0" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 35687 } }
I20260806 02:45:41.195067 8737 raft_consensus.cc:348] T d4b1696a4b98403686a94e1e3abd7d3a P c9e74808dc05428498ef00d1f3d45b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9ec92f5ff745ba9705e2a711320b7b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 34869 } } peers { permanent_uuid: "c9e74808dc05428498ef00d1f3d45b09" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33761 } } peers { permanent_uuid: "6b9fc178691142388ea108e9e5f8a8c0" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 35687 } }
I20260806 02:45:41.195361 8737 raft_consensus.cc:374] T d4b1696a4b98403686a94e1e3abd7d3a P c9e74808dc05428498ef00d1f3d45b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:41.195361 8734 raft_consensus.cc:374] T d4b1696a4b98403686a94e1e3abd7d3a P 6b9fc178691142388ea108e9e5f8a8c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:41.195427 8734 raft_consensus.cc:729] T d4b1696a4b98403686a94e1e3abd7d3a P 6b9fc178691142388ea108e9e5f8a8c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9fc178691142388ea108e9e5f8a8c0, State: Initialized, Role: FOLLOWER
I20260806 02:45:41.195427 8737 raft_consensus.cc:729] T d4b1696a4b98403686a94e1e3abd7d3a P c9e74808dc05428498ef00d1f3d45b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e74808dc05428498ef00d1f3d45b09, State: Initialized, Role: FOLLOWER
I20260806 02:45:41.195858 8312 heartbeater.cc:503] Master 127.1.183.190:33007 was elected leader, sending a full tablet report...
I20260806 02:45:41.195889 8734 consensus_queue.cc:260] T d4b1696a4b98403686a94e1e3abd7d3a P 6b9fc178691142388ea108e9e5f8a8c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9ec92f5ff745ba9705e2a711320b7b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 34869 } } peers { permanent_uuid: "c9e74808dc05428498ef00d1f3d45b09" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33761 } } peers { permanent_uuid: "6b9fc178691142388ea108e9e5f8a8c0" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 35687 } }
I20260806 02:45:41.195889 8737 consensus_queue.cc:260] T d4b1696a4b98403686a94e1e3abd7d3a P c9e74808dc05428498ef00d1f3d45b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9ec92f5ff745ba9705e2a711320b7b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 34869 } } peers { permanent_uuid: "c9e74808dc05428498ef00d1f3d45b09" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33761 } } peers { permanent_uuid: "6b9fc178691142388ea108e9e5f8a8c0" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 35687 } }
I20260806 02:45:41.196297 8736 ts_tablet_manager.cc:1428] T d4b1696a4b98403686a94e1e3abd7d3a P ed9ec92f5ff745ba9705e2a711320b7b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 02:45:41.196902 8737 ts_tablet_manager.cc:1428] T d4b1696a4b98403686a94e1e3abd7d3a P c9e74808dc05428498ef00d1f3d45b09: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260806 02:45:41.196902 8734 ts_tablet_manager.cc:1428] T d4b1696a4b98403686a94e1e3abd7d3a P 6b9fc178691142388ea108e9e5f8a8c0: Time spent starting tablet: real 0.007s user 0.001s sys 0.004s
I20260806 02:45:41.197111 8709 heartbeater.cc:503] Master 127.1.183.190:33007 was elected leader, sending a full tablet report...
I20260806 02:45:41.198077 8501 heartbeater.cc:503] Master 127.1.183.190:33007 was elected leader, sending a full tablet report...
I20260806 02:45:41.201570 8741 raft_consensus.cc:482] T d4b1696a4b98403686a94e1e3abd7d3a P ed9ec92f5ff745ba9705e2a711320b7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:45:41.201783 8741 raft_consensus.cc:504] T d4b1696a4b98403686a94e1e3abd7d3a P ed9ec92f5ff745ba9705e2a711320b7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9ec92f5ff745ba9705e2a711320b7b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 34869 } } peers { permanent_uuid: "c9e74808dc05428498ef00d1f3d45b09" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33761 } } peers { permanent_uuid: "6b9fc178691142388ea108e9e5f8a8c0" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 35687 } }
I20260806 02:45:41.202852 8741 leader_election.cc:288] T d4b1696a4b98403686a94e1e3abd7d3a P ed9ec92f5ff745ba9705e2a711320b7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9e74808dc05428498ef00d1f3d45b09 (127.1.183.130:33761), 6b9fc178691142388ea108e9e5f8a8c0 (127.1.183.131:35687)
I20260806 02:45:41.208791 8731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:41.203848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:57851 (local address 127.1.183.130:33761)
0806 02:45:41.203983 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:41.203988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:41.204000 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:45:41.204040 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:41.204043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:41.204090 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:41.204187 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:41.204602 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:41.205398 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:41.207359 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:41.207998 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:41.208135 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:41.208222 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:41.208567 (+ 345us) server_negotiation.cc:299] Negotiation successful
0806 02:45:41.208654 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:45:41.208823 8733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:41.203588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:42105 (local address 127.1.183.131:35687)
0806 02:45:41.203716 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:41.203722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:41.203900 (+ 178us) server_negotiation.cc:407] Connection header received
0806 02:45:41.203996 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:41.203999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:41.204049 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:41.204134 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:41.204602 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:41.205393 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:41.207359 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:41.208026 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:41.208134 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:41.208222 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:41.208567 (+ 345us) server_negotiation.cc:299] Negotiation successful
0806 02:45:41.208655 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:45:41.208832 8746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:41.203085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:33761 (local address 127.1.183.129:57851)
0806 02:45:41.203378 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:41.203414 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:41.203522 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:41.204201 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:41.204204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:41.204216 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:41.204481 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:41.204488 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:41.205544 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:41.205548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:41.207220 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:41.207227 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:41.208160 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:41.208164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:41.208351 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:41.208423 (+ 72us) client_negotiation.cc:769] Sending connection context
0806 02:45:41.208511 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 02:45:41.208633 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":291,"thread_start_us":110,"threads_started":1}
I20260806 02:45:41.208837 8747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:41.203442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:35687 (local address 127.1.183.129:42105)
0806 02:45:41.203786 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:41.203800 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:41.203910 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:41.204178 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:41.204183 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:41.204202 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:41.204482 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:41.204488 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:41.205544 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:41.205548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:41.207205 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:41.207219 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:41.208196 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:41.208199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:41.208371 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:41.208428 (+ 57us) client_negotiation.cc:769] Sending connection context
0806 02:45:41.208511 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 02:45:41.208634 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":288,"thread_start_us":119,"threads_started":1}
I20260806 02:45:41.209708 8437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b1696a4b98403686a94e1e3abd7d3a" candidate_uuid: "ed9ec92f5ff745ba9705e2a711320b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e74808dc05428498ef00d1f3d45b09" is_pre_election: true
I20260806 02:45:41.209720 8650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b1696a4b98403686a94e1e3abd7d3a" candidate_uuid: "ed9ec92f5ff745ba9705e2a711320b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9fc178691142388ea108e9e5f8a8c0" is_pre_election: true
I20260806 02:45:41.210062 8650 raft_consensus.cc:2454] T d4b1696a4b98403686a94e1e3abd7d3a P 6b9fc178691142388ea108e9e5f8a8c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed9ec92f5ff745ba9705e2a711320b7b in term 0.
I20260806 02:45:41.210071 8437 raft_consensus.cc:2454] T d4b1696a4b98403686a94e1e3abd7d3a P c9e74808dc05428498ef00d1f3d45b09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed9ec92f5ff745ba9705e2a711320b7b in term 0.
I20260806 02:45:41.210565 8148 leader_election.cc:302] T d4b1696a4b98403686a94e1e3abd7d3a P ed9ec92f5ff745ba9705e2a711320b7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b9fc178691142388ea108e9e5f8a8c0, ed9ec92f5ff745ba9705e2a711320b7b; no voters:
I20260806 02:45:41.210884 8741 raft_consensus.cc:2781] T d4b1696a4b98403686a94e1e3abd7d3a P ed9ec92f5ff745ba9705e2a711320b7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:45:41.210981 8741 raft_consensus.cc:482] T d4b1696a4b98403686a94e1e3abd7d3a P ed9ec92f5ff745ba9705e2a711320b7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:45:41.211027 8741 raft_consensus.cc:3037] T d4b1696a4b98403686a94e1e3abd7d3a P ed9ec92f5ff745ba9705e2a711320b7b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:41.212123 8741 raft_consensus.cc:504] T d4b1696a4b98403686a94e1e3abd7d3a P ed9ec92f5ff745ba9705e2a711320b7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9ec92f5ff745ba9705e2a711320b7b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 34869 } } peers { permanent_uuid: "c9e74808dc05428498ef00d1f3d45b09" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33761 } } peers { permanent_uuid: "6b9fc178691142388ea108e9e5f8a8c0" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 35687 } }
I20260806 02:45:41.212574 8741 leader_election.cc:288] T d4b1696a4b98403686a94e1e3abd7d3a P ed9ec92f5ff745ba9705e2a711320b7b [CANDIDATE]: Term 1 election: Requested vote from peers c9e74808dc05428498ef00d1f3d45b09 (127.1.183.130:33761), 6b9fc178691142388ea108e9e5f8a8c0 (127.1.183.131:35687)
I20260806 02:45:41.212946 8437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b1696a4b98403686a94e1e3abd7d3a" candidate_uuid: "ed9ec92f5ff745ba9705e2a711320b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e74808dc05428498ef00d1f3d45b09"
I20260806 02:45:41.213078 8437 raft_consensus.cc:3037] T d4b1696a4b98403686a94e1e3abd7d3a P c9e74808dc05428498ef00d1f3d45b09 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:41.214233 8437 raft_consensus.cc:2454] T d4b1696a4b98403686a94e1e3abd7d3a P c9e74808dc05428498ef00d1f3d45b09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed9ec92f5ff745ba9705e2a711320b7b in term 1.
I20260806 02:45:41.214613 8148 leader_election.cc:302] T d4b1696a4b98403686a94e1e3abd7d3a P ed9ec92f5ff745ba9705e2a711320b7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9e74808dc05428498ef00d1f3d45b09, ed9ec92f5ff745ba9705e2a711320b7b; no voters:
I20260806 02:45:41.214663 8650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b1696a4b98403686a94e1e3abd7d3a" candidate_uuid: "ed9ec92f5ff745ba9705e2a711320b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9fc178691142388ea108e9e5f8a8c0"
I20260806 02:45:41.214815 8741 raft_consensus.cc:2781] T d4b1696a4b98403686a94e1e3abd7d3a P ed9ec92f5ff745ba9705e2a711320b7b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:41.214848 8650 raft_consensus.cc:3037] T d4b1696a4b98403686a94e1e3abd7d3a P 6b9fc178691142388ea108e9e5f8a8c0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:41.215152 8741 raft_consensus.cc:686] T d4b1696a4b98403686a94e1e3abd7d3a P ed9ec92f5ff745ba9705e2a711320b7b [term 1 LEADER]: Becoming Leader. State: Replica: ed9ec92f5ff745ba9705e2a711320b7b, State: Running, Role: LEADER
I20260806 02:45:41.215575 8741 consensus_queue.cc:237] T d4b1696a4b98403686a94e1e3abd7d3a P ed9ec92f5ff745ba9705e2a711320b7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9ec92f5ff745ba9705e2a711320b7b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 34869 } } peers { permanent_uuid: "c9e74808dc05428498ef00d1f3d45b09" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33761 } } peers { permanent_uuid: "6b9fc178691142388ea108e9e5f8a8c0" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 35687 } }
I20260806 02:45:41.216100 8650 raft_consensus.cc:2454] T d4b1696a4b98403686a94e1e3abd7d3a P 6b9fc178691142388ea108e9e5f8a8c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed9ec92f5ff745ba9705e2a711320b7b in term 1.
I20260806 02:45:41.219430 8022 catalog_manager.cc:5495] T d4b1696a4b98403686a94e1e3abd7d3a P ed9ec92f5ff745ba9705e2a711320b7b reported cstate change: term changed from 0 to 1, leader changed from <none> to ed9ec92f5ff745ba9705e2a711320b7b (127.1.183.129). New cstate: current_term: 1 leader_uuid: "ed9ec92f5ff745ba9705e2a711320b7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9ec92f5ff745ba9705e2a711320b7b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 34869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9e74808dc05428498ef00d1f3d45b09" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b9fc178691142388ea108e9e5f8a8c0" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 35687 } health_report { overall_health: UNKNOWN } } }
I20260806 02:45:41.236311 8727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:41.232833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34226 (local address 127.1.183.129:34869)
0806 02:45:41.232975 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:41.232979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:41.232992 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:45:41.233073 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:41.233076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:41.233133 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:41.233223 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:41.233485 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:41.233987 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:41.234856 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:41.235069 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:41.235150 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:41.235421 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:41.235857 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:41.235950 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:41.236014 (+ 64us) server_negotiation.cc:299] Negotiation successful
0806 02:45:41.236129 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:45:41.243162 8650 raft_consensus.cc:1261] T d4b1696a4b98403686a94e1e3abd7d3a P 6b9fc178691142388ea108e9e5f8a8c0 [term 1 FOLLOWER]: Refusing update from remote peer ed9ec92f5ff745ba9705e2a711320b7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:41.243162 8437 raft_consensus.cc:1261] T d4b1696a4b98403686a94e1e3abd7d3a P c9e74808dc05428498ef00d1f3d45b09 [term 1 FOLLOWER]: Refusing update from remote peer ed9ec92f5ff745ba9705e2a711320b7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:41.244028 8741 consensus_queue.cc:1035] T d4b1696a4b98403686a94e1e3abd7d3a P ed9ec92f5ff745ba9705e2a711320b7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9fc178691142388ea108e9e5f8a8c0" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 35687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:41.244357 8748 consensus_queue.cc:1035] T d4b1696a4b98403686a94e1e3abd7d3a P ed9ec92f5ff745ba9705e2a711320b7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e74808dc05428498ef00d1f3d45b09" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:41.254480 8753 mvcc.cc:205] Tried to move back new op lower bound from 7315391861723582464 to 7315391861622824960. Current Snapshot: MvccSnapshot[applied={T|T < 7315391861723582464}]
I20260806 02:45:41.264281 8023 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:41.264547 8023 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:41.267341 8023 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 032c8a141aca4a05a50314d24b6c34ec: Sending DeleteTablet for 3 replicas of tablet d4b1696a4b98403686a94e1e3abd7d3a
I20260806 02:45:41.268290 8215 tablet_service.cc:1514] Processing DeleteTablet for tablet d4b1696a4b98403686a94e1e3abd7d3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:41 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:34212
I20260806 02:45:41.268472 8410 tablet_service.cc:1514] Processing DeleteTablet for tablet d4b1696a4b98403686a94e1e3abd7d3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:41 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:40344
I20260806 02:45:41.268971 8762 tablet_replica.cc:330] T d4b1696a4b98403686a94e1e3abd7d3a P ed9ec92f5ff745ba9705e2a711320b7b: stopping tablet replica
I20260806 02:45:41.269186 8614 tablet_service.cc:1514] Processing DeleteTablet for tablet d4b1696a4b98403686a94e1e3abd7d3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:41 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:40296
I20260806 02:45:41.269469 8763 tablet_replica.cc:330] T d4b1696a4b98403686a94e1e3abd7d3a P c9e74808dc05428498ef00d1f3d45b09: stopping tablet replica
I20260806 02:45:41.269475 8762 raft_consensus.cc:2229] T d4b1696a4b98403686a94e1e3abd7d3a P ed9ec92f5ff745ba9705e2a711320b7b [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:45:41.269838 8763 raft_consensus.cc:2229] T d4b1696a4b98403686a94e1e3abd7d3a P c9e74808dc05428498ef00d1f3d45b09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:41.270088 8762 raft_consensus.cc:2258] T d4b1696a4b98403686a94e1e3abd7d3a P ed9ec92f5ff745ba9705e2a711320b7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:41.270185 8764 tablet_replica.cc:330] T d4b1696a4b98403686a94e1e3abd7d3a P 6b9fc178691142388ea108e9e5f8a8c0: stopping tablet replica
I20260806 02:45:41.270272 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 8120
I20260806 02:45:41.273406 8764 raft_consensus.cc:2229] T d4b1696a4b98403686a94e1e3abd7d3a P 6b9fc178691142388ea108e9e5f8a8c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:41.273770 8764 raft_consensus.cc:2258] T d4b1696a4b98403686a94e1e3abd7d3a P 6b9fc178691142388ea108e9e5f8a8c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:41.278889 8763 raft_consensus.cc:2258] T d4b1696a4b98403686a94e1e3abd7d3a P c9e74808dc05428498ef00d1f3d45b09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:41.281360 8763 ts_tablet_manager.cc:1905] T d4b1696a4b98403686a94e1e3abd7d3a P c9e74808dc05428498ef00d1f3d45b09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:41.281618 8764 ts_tablet_manager.cc:1905] T d4b1696a4b98403686a94e1e3abd7d3a P 6b9fc178691142388ea108e9e5f8a8c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:41.284665 8763 ts_tablet_manager.cc:1918] T d4b1696a4b98403686a94e1e3abd7d3a P c9e74808dc05428498ef00d1f3d45b09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:45:41.284833 8763 log.cc:1196] T d4b1696a4b98403686a94e1e3abd7d3a P c9e74808dc05428498ef00d1f3d45b09: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-1/wal/wals/d4b1696a4b98403686a94e1e3abd7d3a
I20260806 02:45:41.285229 8763 ts_tablet_manager.cc:1939] T d4b1696a4b98403686a94e1e3abd7d3a P c9e74808dc05428498ef00d1f3d45b09: Deleting consensus metadata
I20260806 02:45:41.285632 8764 ts_tablet_manager.cc:1918] T d4b1696a4b98403686a94e1e3abd7d3a P 6b9fc178691142388ea108e9e5f8a8c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:45:41.285766 8764 log.cc:1196] T d4b1696a4b98403686a94e1e3abd7d3a P 6b9fc178691142388ea108e9e5f8a8c0: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984313887056-1758-0/minicluster-data/ts-2/wal/wals/d4b1696a4b98403686a94e1e3abd7d3a
I20260806 02:45:41.286186 8764 ts_tablet_manager.cc:1939] T d4b1696a4b98403686a94e1e3abd7d3a P 6b9fc178691142388ea108e9e5f8a8c0: Deleting consensus metadata
I20260806 02:45:41.286834 8001 catalog_manager.cc:4841] TS c9e74808dc05428498ef00d1f3d45b09 (127.1.183.130:33761): tablet d4b1696a4b98403686a94e1e3abd7d3a (table test-table [id=b8f1af02a5c04f619d1cfc0dc67d0999]) successfully deleted
W20260806 02:45:41.287277 8001 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.183.129:34869 (error 108)
I20260806 02:45:41.287608 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 8319
I20260806 02:45:41.287775 8001 catalog_manager.cc:4841] TS 6b9fc178691142388ea108e9e5f8a8c0 (127.1.183.131:35687): tablet d4b1696a4b98403686a94e1e3abd7d3a (table test-table [id=b8f1af02a5c04f619d1cfc0dc67d0999]) successfully deleted
I20260806 02:45:41.288631 8732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:41.288123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:34869 (local address 127.0.0.1:34228)
0806 02:45:41.288403 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:41.288506 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.129:34869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
W20260806 02:45:41.288883 8001 catalog_manager.cc:4584] TS ed9ec92f5ff745ba9705e2a711320b7b (127.1.183.129:34869): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d4b1696a4b98403686a94e1e3abd7d3a: Network error: Client connection negotiation failed: client connection to 127.1.183.129:34869: connect: Connection refused (error 111)
I20260806 02:45:41.302681 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 8513
I20260806 02:45:41.318444 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 7978
2026-08-06T02:45:41Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2614 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-06T02:45:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:41Z Disabled control of system clock
I20260806 02:45:41.358656 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:36463
--webserver_interface=127.1.183.190
--webserver_port=0
--builtin_ntp_servers=127.1.183.148:43607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:36463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 02:45:41.481241 8770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:41.481557 8770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:41.481597 8770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:41.485932 8770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:41.486006 8770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:41.486022 8770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:41.486042 8770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:41.486057 8770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:41.491215 8770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:43607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:36463
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:36463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:41.492488 8770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:41.493996 8770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:41.502954 8777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:41.502966 8778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:41.503026 8780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:41.503218 8770 server_base.cc:1034] running on GCE node
I20260806 02:45:41.503981 8770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:41.505285 8770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:41.506526 8770 hybrid_clock.cc:648] HybridClock initialized: now 1785984341506509 us; error 33 us; skew 500 ppm
I20260806 02:45:41.508800 8770 webserver.cc:460] Webserver started at http://127.1.183.190:39861/ using document root <none> and password file <none>
I20260806 02:45:41.509436 8770 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:41.509532 8770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:41.509817 8770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:41.512030 8770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "d2319e2c1ca8406e9040bc22f9577d55"
format_stamp: "Formatted at 2026-08-06 02:45:41 on dist-test-slave-gmjp"
server_key: "65de92b2da31f0a4e8ed450de2b6e344"
server_key_iv: "35ef62ecc2430714b0561b97ddd942a0"
server_key_version: "encryptionkey@0"
I20260806 02:45:41.512691 8770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "d2319e2c1ca8406e9040bc22f9577d55"
format_stamp: "Formatted at 2026-08-06 02:45:41 on dist-test-slave-gmjp"
server_key: "65de92b2da31f0a4e8ed450de2b6e344"
server_key_iv: "35ef62ecc2430714b0561b97ddd942a0"
server_key_version: "encryptionkey@0"
I20260806 02:45:41.516652 8770 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:45:41.519263 8789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:41.520623 8770 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:45:41.520774 8770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "d2319e2c1ca8406e9040bc22f9577d55"
format_stamp: "Formatted at 2026-08-06 02:45:41 on dist-test-slave-gmjp"
server_key: "65de92b2da31f0a4e8ed450de2b6e344"
server_key_iv: "35ef62ecc2430714b0561b97ddd942a0"
server_key_version: "encryptionkey@0"
I20260806 02:45:41.520905 8770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:41.525713 8770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:41.526391 8770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:41.526584 8770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:41.533913 8770 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:36463
I20260806 02:45:41.533923 8869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:36463 every 8 connection(s)
I20260806 02:45:41.534942 8770 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:41.535465 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 8770
I20260806 02:45:41.535610 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:45:41.535933 1758 external_mini_cluster.cc:1428] Setting key 4ff4b898f01bda8ec2c76f27c89cc96e
I20260806 02:45:41.538322 8870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:41.542379 8873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:41.537386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54792 (local address 127.1.183.190:36463)
0806 02:45:41.538580 (+ 1194us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:41.538597 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:41.538636 (+ 39us) server_negotiation.cc:407] Connection header received
0806 02:45:41.539306 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:41.539322 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:41.539693 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:41.540023 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:41.541009 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:41.541014 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:41.541026 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:41.541036 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:41.541092 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:41.541109 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:41.541230 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:41.541344 (+ 114us) server_negotiation.cc:299] Negotiation successful
0806 02:45:41.541625 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":585,"threads_started":1}
I20260806 02:45:41.545256 8870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2319e2c1ca8406e9040bc22f9577d55: Bootstrap starting.
I20260806 02:45:41.547595 8870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2319e2c1ca8406e9040bc22f9577d55: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:41.548478 8870 log.cc:824] T 00000000000000000000000000000000 P d2319e2c1ca8406e9040bc22f9577d55: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:41.550626 8870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2319e2c1ca8406e9040bc22f9577d55: No bootstrap required, opened a new log
I20260806 02:45:41.554461 8870 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2319e2c1ca8406e9040bc22f9577d55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2319e2c1ca8406e9040bc22f9577d55" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 36463 } }
I20260806 02:45:41.554771 8870 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2319e2c1ca8406e9040bc22f9577d55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:41.554869 8870 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2319e2c1ca8406e9040bc22f9577d55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2319e2c1ca8406e9040bc22f9577d55, State: Initialized, Role: FOLLOWER
I20260806 02:45:41.555352 8870 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2319e2c1ca8406e9040bc22f9577d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2319e2c1ca8406e9040bc22f9577d55" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 36463 } }
I20260806 02:45:41.555517 8870 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2319e2c1ca8406e9040bc22f9577d55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:41.555608 8870 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2319e2c1ca8406e9040bc22f9577d55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:41.555738 8870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2319e2c1ca8406e9040bc22f9577d55 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:41.557041 8870 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2319e2c1ca8406e9040bc22f9577d55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2319e2c1ca8406e9040bc22f9577d55" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 36463 } }
I20260806 02:45:41.557442 8870 leader_election.cc:302] T 00000000000000000000000000000000 P d2319e2c1ca8406e9040bc22f9577d55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2319e2c1ca8406e9040bc22f9577d55; no voters:
I20260806 02:45:41.557819 8870 leader_election.cc:288] T 00000000000000000000000000000000 P d2319e2c1ca8406e9040bc22f9577d55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:41.557962 8876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2319e2c1ca8406e9040bc22f9577d55 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:41.558223 8876 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2319e2c1ca8406e9040bc22f9577d55 [term 1 LEADER]: Becoming Leader. State: Replica: d2319e2c1ca8406e9040bc22f9577d55, State: Running, Role: LEADER
I20260806 02:45:41.558576 8876 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2319e2c1ca8406e9040bc22f9577d55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2319e2c1ca8406e9040bc22f9577d55" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 36463 } }
I20260806 02:45:41.559264 8870 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2319e2c1ca8406e9040bc22f9577d55 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:41.560531 8880 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2319e2c1ca8406e9040bc22f9577d55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2319e2c1ca8406e9040bc22f9577d55. Latest consensus state: current_term: 1 leader_uuid: "d2319e2c1ca8406e9040bc22f9577d55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2319e2c1ca8406e9040bc22f9577d55" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 36463 } } }
I20260806 02:45:41.560676 8880 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2319e2c1ca8406e9040bc22f9577d55 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:41.561110 8889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:41.561703 8878 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2319e2c1ca8406e9040bc22f9577d55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2319e2c1ca8406e9040bc22f9577d55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2319e2c1ca8406e9040bc22f9577d55" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 36463 } } }
I20260806 02:45:41.561851 8878 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2319e2c1ca8406e9040bc22f9577d55 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:41.564224 8889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:41.570439 8889 catalog_manager.cc:1347] Generated new cluster ID: a480232f2cec4b7b959f8eed6b57eafb
I20260806 02:45:41.570551 8889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:41.585049 8889 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:41.585482 8889 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:41.593879 8889 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2319e2c1ca8406e9040bc22f9577d55: Generated new TSK 0
I20260806 02:45:41.594823 8889 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:45:41.598865 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.129:0
--local_ip_for_outbound_sockets=127.1.183.129
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:36463
--builtin_ntp_servers=127.1.183.148:43607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 02:45:41.722424 8902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:41.722786 8902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:41.722864 8902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:41.727460 8902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:41.727559 8902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:41.727655 8902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.129
I20260806 02:45:41.733386 8902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:43607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:36463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.129
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:41.734619 8902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:41.736191 8902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:41.746309 8908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:41.746309 8912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:41.746336 8909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:41.746824 8902 server_base.cc:1034] running on GCE node
I20260806 02:45:41.747341 8902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:41.748059 8902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:41.749250 8902 hybrid_clock.cc:648] HybridClock initialized: now 1785984341749214 us; error 48 us; skew 500 ppm
I20260806 02:45:41.751299 8902 webserver.cc:460] Webserver started at http://127.1.183.129:43615/ using document root <none> and password file <none>
I20260806 02:45:41.751933 8902 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:41.752036 8902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:41.752338 8902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:41.754546 8902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-0/data/instance:
uuid: "237b3fcfab304c2c9dc95487d3968dce"
format_stamp: "Formatted at 2026-08-06 02:45:41 on dist-test-slave-gmjp"
server_key: "e72975c41b6d6603021bb9f1b4cc7d4b"
server_key_iv: "c9852e31a3e3d41d01513c5448e6ee7d"
server_key_version: "encryptionkey@0"
I20260806 02:45:41.755151 8902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance:
uuid: "237b3fcfab304c2c9dc95487d3968dce"
format_stamp: "Formatted at 2026-08-06 02:45:41 on dist-test-slave-gmjp"
server_key: "e72975c41b6d6603021bb9f1b4cc7d4b"
server_key_iv: "c9852e31a3e3d41d01513c5448e6ee7d"
server_key_version: "encryptionkey@0"
I20260806 02:45:41.759419 8902 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 02:45:41.762341 8920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:41.763697 8902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:45:41.763865 8902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-0/wal
uuid: "237b3fcfab304c2c9dc95487d3968dce"
format_stamp: "Formatted at 2026-08-06 02:45:41 on dist-test-slave-gmjp"
server_key: "e72975c41b6d6603021bb9f1b4cc7d4b"
server_key_iv: "c9852e31a3e3d41d01513c5448e6ee7d"
server_key_version: "encryptionkey@0"
I20260806 02:45:41.764006 8902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:41.768922 8902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:41.769574 8902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:41.769780 8902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:41.770368 8902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:41.771345 8902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:41.771417 8902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:41.771489 8902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:41.771538 8902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:41.781069 8902 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.129:36885
I20260806 02:45:41.781088 9081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.129:36885 every 8 connection(s)
I20260806 02:45:41.782088 8902 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
I20260806 02:45:41.787040 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 8902
I20260806 02:45:41.787209 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance
I20260806 02:45:41.787503 1758 external_mini_cluster.cc:1428] Setting key cd035fee31474c29283193db9ee65761
I20260806 02:45:41.789407 8873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:41.784073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:33697 (local address 127.1.183.190:36463)
0806 02:45:41.784296 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:41.784302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:41.785510 (+ 1208us) server_negotiation.cc:407] Connection header received
0806 02:45:41.787011 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:41.787018 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:41.787073 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:41.787201 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:41.788745 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:41.788749 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:41.788752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:41.788773 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:41.788795 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:41.788822 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:41.788951 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:41.789215 (+ 264us) server_negotiation.cc:299] Negotiation successful
0806 02:45:41.789268 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260806 02:45:41.789836 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.130:0
--local_ip_for_outbound_sockets=127.1.183.130
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:36463
--builtin_ntp_servers=127.1.183.148:43607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 02:45:41.790459 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:41.784573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:36463 (local address 127.1.183.129:33697)
0806 02:45:41.785321 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:41.785368 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:41.786750 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:41.787487 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:41.787495 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:41.787651 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:41.788447 (+ 796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:45:41.788449 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:41.788505 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:45:41.788584 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:41.789062 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:41.789071 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:41.789076 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 02:45:41.789265 (+ 189us) client_negotiation.cc:240] Negotiation successful
0806 02:45:41.789552 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":110,"threads_started":1}
I20260806 02:45:41.791887 9082 heartbeater.cc:347] Connected to a master server at 127.1.183.190:36463
I20260806 02:45:41.792196 9082 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:41.792745 9082 heartbeater.cc:511] Master 127.1.183.190:36463 requested a full tablet report, sending...
I20260806 02:45:41.794699 8821 ts_manager.cc:194] Registered new tserver with Master: 237b3fcfab304c2c9dc95487d3968dce (127.1.183.129:36885)
W20260806 02:45:41.912904 9086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:41.913213 9086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:41.913257 9086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:41.917501 9086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:41.917580 9086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:41.917641 9086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.130
I20260806 02:45:41.922839 9086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:43607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:36463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.130
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:41.924028 9086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:41.925668 9086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:41.935983 9100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:41.935974 9098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:41.936074 9086 server_base.cc:1034] running on GCE node
W20260806 02:45:41.936080 9097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:41.936753 9086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:41.937543 9086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:41.938745 9086 hybrid_clock.cc:648] HybridClock initialized: now 1785984341938716 us; error 39 us; skew 500 ppm
I20260806 02:45:41.940790 9086 webserver.cc:460] Webserver started at http://127.1.183.130:37207/ using document root <none> and password file <none>
I20260806 02:45:41.941423 9086 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:41.941529 9086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:41.941776 9086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:41.943933 9086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-1/data/instance:
uuid: "edf2c469841842bebe56cab2c925b669"
format_stamp: "Formatted at 2026-08-06 02:45:41 on dist-test-slave-gmjp"
server_key: "fbb6fc88f983cbbce9ea88fc15e10ab0"
server_key_iv: "5a4b2926d164df4e1e05b465c69b856a"
server_key_version: "encryptionkey@0"
I20260806 02:45:41.944592 9086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance:
uuid: "edf2c469841842bebe56cab2c925b669"
format_stamp: "Formatted at 2026-08-06 02:45:41 on dist-test-slave-gmjp"
server_key: "fbb6fc88f983cbbce9ea88fc15e10ab0"
server_key_iv: "5a4b2926d164df4e1e05b465c69b856a"
server_key_version: "encryptionkey@0"
I20260806 02:45:41.948566 9086 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:45:41.951100 9109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:41.952447 9086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:45:41.952584 9086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-1/wal
uuid: "edf2c469841842bebe56cab2c925b669"
format_stamp: "Formatted at 2026-08-06 02:45:41 on dist-test-slave-gmjp"
server_key: "fbb6fc88f983cbbce9ea88fc15e10ab0"
server_key_iv: "5a4b2926d164df4e1e05b465c69b856a"
server_key_version: "encryptionkey@0"
I20260806 02:45:41.952706 9086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:41.957523 9086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:41.958184 9086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:41.958384 9086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:41.959005 9086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:41.960071 9086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:41.960139 9086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:41.960230 9086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:41.960274 9086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:41.970038 9086 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.130:37903
I20260806 02:45:41.970069 9273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.130:37903 every 8 connection(s)
I20260806 02:45:41.971004 9086 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
I20260806 02:45:41.977347 8873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:41.972905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:43157 (local address 127.1.183.190:36463)
0806 02:45:41.973056 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:41.973060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:41.974105 (+ 1045us) server_negotiation.cc:407] Connection header received
0806 02:45:41.975470 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:41.975473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:41.975521 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:41.975635 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:41.976833 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:41.976837 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:41.976839 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:41.976861 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:41.976890 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:41.976898 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:41.976998 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:41.977134 (+ 136us) server_negotiation.cc:299] Negotiation successful
0806 02:45:41.977187 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:45:41.978217 9281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:41.973259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:36463 (local address 127.1.183.130:43157)
0806 02:45:41.973945 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:41.973986 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:41.975233 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:41.975811 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:41.975818 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:41.975919 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:41.976566 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:45:41.976568 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:41.976609 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:45:41.976678 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:41.976980 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:41.976990 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:41.976995 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 02:45:41.977182 (+ 187us) client_negotiation.cc:240] Negotiation successful
0806 02:45:41.977462 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":122,"threads_started":1}
I20260806 02:45:41.979182 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 9086
I20260806 02:45:41.979302 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance
I20260806 02:45:41.979530 9275 heartbeater.cc:347] Connected to a master server at 127.1.183.190:36463
I20260806 02:45:41.979665 1758 external_mini_cluster.cc:1428] Setting key d19cd6a2d3a9e196c3c0a2d63fcb209a
I20260806 02:45:41.979792 9275 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:41.980324 9275 heartbeater.cc:511] Master 127.1.183.190:36463 requested a full tablet report, sending...
I20260806 02:45:41.981372 8821 ts_manager.cc:194] Registered new tserver with Master: edf2c469841842bebe56cab2c925b669 (127.1.183.130:37903)
I20260806 02:45:41.981750 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.131:0
--local_ip_for_outbound_sockets=127.1.183.131
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:36463
--builtin_ntp_servers=127.1.183.148:43607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 02:45:42.106262 9282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:42.106634 9282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:42.106753 9282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:42.111256 9282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:42.111369 9282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:42.111482 9282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.131
I20260806 02:45:42.116720 9282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:43607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:36463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.131
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:42.118022 9282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:42.119626 9282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:45:42.130676 9282 server_base.cc:1034] running on GCE node
W20260806 02:45:42.130620 9292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:42.130620 9295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:42.130633 9291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:42.131340 9282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:42.132061 9282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:42.133332 9282 hybrid_clock.cc:648] HybridClock initialized: now 1785984342133266 us; error 30 us; skew 500 ppm
I20260806 02:45:42.135305 9282 webserver.cc:460] Webserver started at http://127.1.183.131:42441/ using document root <none> and password file <none>
I20260806 02:45:42.135950 9282 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:42.136049 9282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:42.136337 9282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:42.138571 9282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-2/data/instance:
uuid: "8eaa347195e04db2a66afc180fd6de79"
format_stamp: "Formatted at 2026-08-06 02:45:42 on dist-test-slave-gmjp"
server_key: "11835ad1781da34900e3b45b7c1c8618"
server_key_iv: "f431530672672b3fae5779bee3ce58e7"
server_key_version: "encryptionkey@0"
I20260806 02:45:42.139228 9282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance:
uuid: "8eaa347195e04db2a66afc180fd6de79"
format_stamp: "Formatted at 2026-08-06 02:45:42 on dist-test-slave-gmjp"
server_key: "11835ad1781da34900e3b45b7c1c8618"
server_key_iv: "f431530672672b3fae5779bee3ce58e7"
server_key_version: "encryptionkey@0"
I20260806 02:45:42.143525 9282 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 02:45:42.146196 9302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:42.147514 9282 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:45:42.147665 9282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-2/wal
uuid: "8eaa347195e04db2a66afc180fd6de79"
format_stamp: "Formatted at 2026-08-06 02:45:42 on dist-test-slave-gmjp"
server_key: "11835ad1781da34900e3b45b7c1c8618"
server_key_iv: "f431530672672b3fae5779bee3ce58e7"
server_key_version: "encryptionkey@0"
I20260806 02:45:42.147791 9282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:42.152797 9282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:42.153468 9282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:42.153659 9282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:42.154233 9282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:42.155190 9282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:42.155262 9282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:42.155330 9282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:42.155385 9282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:42.165397 9282 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.131:37775
I20260806 02:45:42.165434 9478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.131:37775 every 8 connection(s)
I20260806 02:45:42.166364 9282 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
I20260806 02:45:42.169567 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 9282
I20260806 02:45:42.169708 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance
I20260806 02:45:42.170053 1758 external_mini_cluster.cc:1428] Setting key 3ba970fb523789632ac99e715636ac32
I20260806 02:45:42.173488 8873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:42.168485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:34717 (local address 127.1.183.190:36463)
0806 02:45:42.168665 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:42.168669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:42.170095 (+ 1426us) server_negotiation.cc:407] Connection header received
0806 02:45:42.171727 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:42.171730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:42.171771 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:42.171829 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:42.173095 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:42.173097 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:42.173100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:42.173111 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:42.173130 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:42.173138 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:42.173209 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:42.173337 (+ 128us) server_negotiation.cc:299] Negotiation successful
0806 02:45:42.173374 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 02:45:42.174443 9486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:42.169006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:36463 (local address 127.1.183.131:34717)
0806 02:45:42.169899 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:42.169961 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:42.171498 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:42.172013 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:42.172022 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:42.172145 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:42.172846 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:45:42.172848 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:42.172892 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:45:42.172964 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:42.173216 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:42.173223 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:42.173227 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 02:45:42.173377 (+ 150us) client_negotiation.cc:240] Negotiation successful
0806 02:45:42.173579 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":132,"threads_started":1}
I20260806 02:45:42.175554 9479 heartbeater.cc:347] Connected to a master server at 127.1.183.190:36463
I20260806 02:45:42.175819 9479 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:42.176331 9479 heartbeater.cc:511] Master 127.1.183.190:36463 requested a full tablet report, sending...
I20260806 02:45:42.177418 8821 ts_manager.cc:194] Registered new tserver with Master: 8eaa347195e04db2a66afc180fd6de79 (127.1.183.131:37775)
I20260806 02:45:42.184546 1758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:45:42.188423 8873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:42.186585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54802 (local address 127.1.183.190:36463)
0806 02:45:42.186721 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:42.186732 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:42.186957 (+ 225us) server_negotiation.cc:407] Connection header received
0806 02:45:42.187048 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:42.187050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:42.187083 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:42.187152 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:42.187919 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:42.187922 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:42.187924 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:42.187935 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:42.187953 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:42.187962 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:42.188032 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:42.188273 (+ 241us) server_negotiation.cc:299] Negotiation successful
0806 02:45:42.188309 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:45:42.191462 8821 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:45:42.194829 8821 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:45:42.207978 9499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:42.204585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38464 (local address 127.1.183.131:37775)
0806 02:45:42.205540 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:42.205554 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:42.205580 (+ 26us) server_negotiation.cc:407] Connection header received
0806 02:45:42.205637 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:42.205642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:42.205854 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:42.206006 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:42.207368 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:42.207376 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:42.207388 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:42.207401 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:42.207447 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:42.207470 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:42.207599 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:42.207728 (+ 129us) server_negotiation.cc:299] Negotiation successful
0806 02:45:42.207807 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":845,"thread_start_us":142,"threads_started":1}
I20260806 02:45:42.207980 9497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:42.205234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:36885 (local address 127.0.0.1:46962)
0806 02:45:42.206064 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:42.206078 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:42.206142 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:42.206659 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:42.206662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:42.206679 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:42.207145 (+ 466us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:45:42.207147 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:42.207159 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:45:42.207226 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:42.207670 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:42.207674 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:42.207677 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 02:45:42.207762 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 02:45:42.207804 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"thread_start_us":100,"threads_started":1}
I20260806 02:45:42.208534 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:42.204758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47202 (local address 127.1.183.130:37903)
0806 02:45:42.205320 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:42.205336 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:42.205913 (+ 577us) server_negotiation.cc:407] Connection header received
0806 02:45:42.205977 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:42.205983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:42.206156 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:42.206313 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:42.207584 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:42.207590 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:42.207599 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:42.207614 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:42.207671 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:42.207714 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:42.207812 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:42.208046 (+ 234us) server_negotiation.cc:299] Negotiation successful
0806 02:45:42.208360 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":160,"threads_started":1}
I20260806 02:45:42.208618 9504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:42.205678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46962 (local address 127.1.183.129:36885)
0806 02:45:42.206160 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:42.206171 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:42.206197 (+ 26us) server_negotiation.cc:407] Connection header received
0806 02:45:42.206252 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:42.206260 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:42.206423 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:42.206549 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:42.207413 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:42.207420 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:42.207431 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:42.207445 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:42.207492 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:42.207529 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:42.207631 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:42.208379 (+ 748us) server_negotiation.cc:299] Negotiation successful
0806 02:45:42.208450 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":113,"threads_started":1}
I20260806 02:45:42.208638 9498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:42.204475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:37775 (local address 127.0.0.1:38464)
0806 02:45:42.205323 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:42.205381 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:42.205491 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:42.206266 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:42.206272 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:42.206350 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:42.207125 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:45:42.207128 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:42.207151 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:45:42.207228 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:42.207570 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:42.207583 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:42.207592 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 02:45:42.207704 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 02:45:42.207775 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"spinlock_wait_cycles":9984,"thread_start_us":176,"threads_started":1}
I20260806 02:45:42.209591 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:42.204496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:37903 (local address 127.0.0.1:47202)
0806 02:45:42.205711 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:42.205726 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:42.205814 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:42.206902 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:42.206906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:42.206922 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:42.207412 (+ 490us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:45:42.207414 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:42.207423 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:45:42.207446 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:42.207853 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:42.207857 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:42.207860 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 02:45:42.207928 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 02:45:42.207956 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1152,"thread_start_us":87,"threads_started":1}
I20260806 02:45:42.211942 8989 tablet_service.cc:1467] Processing CreateTablet for tablet 220f79826f574e2c9cca29432049d453 (DEFAULT_TABLE table=test-table [id=4234a1d8530943e7b792ff7f37c0aad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:42.211943 9381 tablet_service.cc:1467] Processing CreateTablet for tablet 220f79826f574e2c9cca29432049d453 (DEFAULT_TABLE table=test-table [id=4234a1d8530943e7b792ff7f37c0aad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:42.212106 9181 tablet_service.cc:1467] Processing CreateTablet for tablet 220f79826f574e2c9cca29432049d453 (DEFAULT_TABLE table=test-table [id=4234a1d8530943e7b792ff7f37c0aad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:42.213174 9181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 220f79826f574e2c9cca29432049d453. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:42.213179 8989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 220f79826f574e2c9cca29432049d453. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:42.213181 9381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 220f79826f574e2c9cca29432049d453. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:42.219756 9505 tablet_bootstrap.cc:492] T 220f79826f574e2c9cca29432049d453 P edf2c469841842bebe56cab2c925b669: Bootstrap starting.
I20260806 02:45:42.221206 9506 tablet_bootstrap.cc:492] T 220f79826f574e2c9cca29432049d453 P 8eaa347195e04db2a66afc180fd6de79: Bootstrap starting.
I20260806 02:45:42.221585 9508 tablet_bootstrap.cc:492] T 220f79826f574e2c9cca29432049d453 P 237b3fcfab304c2c9dc95487d3968dce: Bootstrap starting.
I20260806 02:45:42.223162 9505 tablet_bootstrap.cc:654] T 220f79826f574e2c9cca29432049d453 P edf2c469841842bebe56cab2c925b669: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:42.223179 9506 tablet_bootstrap.cc:654] T 220f79826f574e2c9cca29432049d453 P 8eaa347195e04db2a66afc180fd6de79: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:42.223812 9508 tablet_bootstrap.cc:654] T 220f79826f574e2c9cca29432049d453 P 237b3fcfab304c2c9dc95487d3968dce: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:42.224030 9506 log.cc:824] T 220f79826f574e2c9cca29432049d453 P 8eaa347195e04db2a66afc180fd6de79: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:42.224042 9505 log.cc:824] T 220f79826f574e2c9cca29432049d453 P edf2c469841842bebe56cab2c925b669: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:42.224694 9508 log.cc:824] T 220f79826f574e2c9cca29432049d453 P 237b3fcfab304c2c9dc95487d3968dce: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:42.226401 9506 tablet_bootstrap.cc:492] T 220f79826f574e2c9cca29432049d453 P 8eaa347195e04db2a66afc180fd6de79: No bootstrap required, opened a new log
I20260806 02:45:42.226401 9505 tablet_bootstrap.cc:492] T 220f79826f574e2c9cca29432049d453 P edf2c469841842bebe56cab2c925b669: No bootstrap required, opened a new log
I20260806 02:45:42.226552 9508 tablet_bootstrap.cc:492] T 220f79826f574e2c9cca29432049d453 P 237b3fcfab304c2c9dc95487d3968dce: No bootstrap required, opened a new log
I20260806 02:45:42.226619 9506 ts_tablet_manager.cc:1397] T 220f79826f574e2c9cca29432049d453 P 8eaa347195e04db2a66afc180fd6de79: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:45:42.226619 9505 ts_tablet_manager.cc:1397] T 220f79826f574e2c9cca29432049d453 P edf2c469841842bebe56cab2c925b669: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260806 02:45:42.226819 9508 ts_tablet_manager.cc:1397] T 220f79826f574e2c9cca29432049d453 P 237b3fcfab304c2c9dc95487d3968dce: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:45:42.230454 9508 raft_consensus.cc:348] T 220f79826f574e2c9cca29432049d453 P 237b3fcfab304c2c9dc95487d3968dce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaa347195e04db2a66afc180fd6de79" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37775 } } peers { permanent_uuid: "edf2c469841842bebe56cab2c925b669" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37903 } } peers { permanent_uuid: "237b3fcfab304c2c9dc95487d3968dce" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 36885 } }
I20260806 02:45:42.230718 9508 raft_consensus.cc:374] T 220f79826f574e2c9cca29432049d453 P 237b3fcfab304c2c9dc95487d3968dce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:42.230773 9508 raft_consensus.cc:729] T 220f79826f574e2c9cca29432049d453 P 237b3fcfab304c2c9dc95487d3968dce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 237b3fcfab304c2c9dc95487d3968dce, State: Initialized, Role: FOLLOWER
I20260806 02:45:42.231303 9508 consensus_queue.cc:260] T 220f79826f574e2c9cca29432049d453 P 237b3fcfab304c2c9dc95487d3968dce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaa347195e04db2a66afc180fd6de79" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37775 } } peers { permanent_uuid: "edf2c469841842bebe56cab2c925b669" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37903 } } peers { permanent_uuid: "237b3fcfab304c2c9dc95487d3968dce" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 36885 } }
I20260806 02:45:42.231362 9505 raft_consensus.cc:348] T 220f79826f574e2c9cca29432049d453 P edf2c469841842bebe56cab2c925b669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaa347195e04db2a66afc180fd6de79" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37775 } } peers { permanent_uuid: "edf2c469841842bebe56cab2c925b669" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37903 } } peers { permanent_uuid: "237b3fcfab304c2c9dc95487d3968dce" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 36885 } }
I20260806 02:45:42.231369 9506 raft_consensus.cc:348] T 220f79826f574e2c9cca29432049d453 P 8eaa347195e04db2a66afc180fd6de79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaa347195e04db2a66afc180fd6de79" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37775 } } peers { permanent_uuid: "edf2c469841842bebe56cab2c925b669" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37903 } } peers { permanent_uuid: "237b3fcfab304c2c9dc95487d3968dce" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 36885 } }
I20260806 02:45:42.231619 9505 raft_consensus.cc:374] T 220f79826f574e2c9cca29432049d453 P edf2c469841842bebe56cab2c925b669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:42.231619 9506 raft_consensus.cc:374] T 220f79826f574e2c9cca29432049d453 P 8eaa347195e04db2a66afc180fd6de79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:42.231670 9505 raft_consensus.cc:729] T 220f79826f574e2c9cca29432049d453 P edf2c469841842bebe56cab2c925b669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edf2c469841842bebe56cab2c925b669, State: Initialized, Role: FOLLOWER
I20260806 02:45:42.231681 9506 raft_consensus.cc:729] T 220f79826f574e2c9cca29432049d453 P 8eaa347195e04db2a66afc180fd6de79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eaa347195e04db2a66afc180fd6de79, State: Initialized, Role: FOLLOWER
I20260806 02:45:42.232214 9082 heartbeater.cc:503] Master 127.1.183.190:36463 was elected leader, sending a full tablet report...
I20260806 02:45:42.232189 9506 consensus_queue.cc:260] T 220f79826f574e2c9cca29432049d453 P 8eaa347195e04db2a66afc180fd6de79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaa347195e04db2a66afc180fd6de79" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37775 } } peers { permanent_uuid: "edf2c469841842bebe56cab2c925b669" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37903 } } peers { permanent_uuid: "237b3fcfab304c2c9dc95487d3968dce" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 36885 } }
I20260806 02:45:42.232194 9505 consensus_queue.cc:260] T 220f79826f574e2c9cca29432049d453 P edf2c469841842bebe56cab2c925b669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaa347195e04db2a66afc180fd6de79" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37775 } } peers { permanent_uuid: "edf2c469841842bebe56cab2c925b669" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37903 } } peers { permanent_uuid: "237b3fcfab304c2c9dc95487d3968dce" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 36885 } }
I20260806 02:45:42.232802 9508 ts_tablet_manager.cc:1428] T 220f79826f574e2c9cca29432049d453 P 237b3fcfab304c2c9dc95487d3968dce: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:45:42.233184 9506 ts_tablet_manager.cc:1428] T 220f79826f574e2c9cca29432049d453 P 8eaa347195e04db2a66afc180fd6de79: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 02:45:42.233184 9505 ts_tablet_manager.cc:1428] T 220f79826f574e2c9cca29432049d453 P edf2c469841842bebe56cab2c925b669: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260806 02:45:42.233525 9275 heartbeater.cc:503] Master 127.1.183.190:36463 was elected leader, sending a full tablet report...
I20260806 02:45:42.233554 9479 heartbeater.cc:503] Master 127.1.183.190:36463 was elected leader, sending a full tablet report...
I20260806 02:45:42.274760 9514 raft_consensus.cc:482] T 220f79826f574e2c9cca29432049d453 P 237b3fcfab304c2c9dc95487d3968dce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:45:42.275038 9514 raft_consensus.cc:504] T 220f79826f574e2c9cca29432049d453 P 237b3fcfab304c2c9dc95487d3968dce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaa347195e04db2a66afc180fd6de79" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37775 } } peers { permanent_uuid: "edf2c469841842bebe56cab2c925b669" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37903 } } peers { permanent_uuid: "237b3fcfab304c2c9dc95487d3968dce" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 36885 } }
I20260806 02:45:42.276489 9514 leader_election.cc:288] T 220f79826f574e2c9cca29432049d453 P 237b3fcfab304c2c9dc95487d3968dce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8eaa347195e04db2a66afc180fd6de79 (127.1.183.131:37775), edf2c469841842bebe56cab2c925b669 (127.1.183.130:37903)
I20260806 02:45:42.278563 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:42.276778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:37903 (local address 127.1.183.129:46811)
0806 02:45:42.276921 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:42.276938 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:42.277032 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:42.277316 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:42.277320 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:42.277335 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:42.277888 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:45:42.277891 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:42.277905 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:45:42.277940 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:42.278261 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:42.278264 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:42.278268 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 02:45:42.278358 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 02:45:42.278399 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260806 02:45:42.278620 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:42.276946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:46811 (local address 127.1.183.130:37903)
0806 02:45:42.277076 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:42.277082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:42.277099 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:45:42.277146 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:42.277149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:42.277191 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:42.277288 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:42.278102 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:42.278106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:42.278109 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:42.278123 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:42.278144 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:42.278157 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:42.278259 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:42.278369 (+ 110us) server_negotiation.cc:299] Negotiation successful
0806 02:45:42.278441 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:45:42.278745 9499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:42.276906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:42771 (local address 127.1.183.131:37775)
0806 02:45:42.277049 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:42.277056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:42.277366 (+ 310us) server_negotiation.cc:407] Connection header received
0806 02:45:42.277507 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:42.277515 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:42.277550 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:42.277636 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:42.278294 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:42.278301 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:42.278307 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:42.278325 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:42.278351 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:42.278363 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:42.278464 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:42.278568 (+ 104us) server_negotiation.cc:299] Negotiation successful
0806 02:45:42.278615 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:45:42.278756 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:42.276795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:37775 (local address 127.1.183.129:42771)
0806 02:45:42.277259 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:42.277271 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:42.277387 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:42.277653 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:42.277656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:42.277670 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:42.278090 (+ 420us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:45:42.278092 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:42.278103 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:45:42.278130 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:42.278470 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:42.278474 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:42.278477 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 02:45:42.278569 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 02:45:42.278603 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":124,"thread_start_us":122,"threads_started":1}
I20260806 02:45:42.279464 9215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220f79826f574e2c9cca29432049d453" candidate_uuid: "237b3fcfab304c2c9dc95487d3968dce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edf2c469841842bebe56cab2c925b669" is_pre_election: true
I20260806 02:45:42.279472 9414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220f79826f574e2c9cca29432049d453" candidate_uuid: "237b3fcfab304c2c9dc95487d3968dce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eaa347195e04db2a66afc180fd6de79" is_pre_election: true
I20260806 02:45:42.279778 9215 raft_consensus.cc:2454] T 220f79826f574e2c9cca29432049d453 P edf2c469841842bebe56cab2c925b669 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 237b3fcfab304c2c9dc95487d3968dce in term 0.
I20260806 02:45:42.279778 9414 raft_consensus.cc:2454] T 220f79826f574e2c9cca29432049d453 P 8eaa347195e04db2a66afc180fd6de79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 237b3fcfab304c2c9dc95487d3968dce in term 0.
I20260806 02:45:42.280339 8922 leader_election.cc:302] T 220f79826f574e2c9cca29432049d453 P 237b3fcfab304c2c9dc95487d3968dce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 237b3fcfab304c2c9dc95487d3968dce, 8eaa347195e04db2a66afc180fd6de79, edf2c469841842bebe56cab2c925b669; no voters:
I20260806 02:45:42.280638 9514 raft_consensus.cc:2781] T 220f79826f574e2c9cca29432049d453 P 237b3fcfab304c2c9dc95487d3968dce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:45:42.280719 9514 raft_consensus.cc:482] T 220f79826f574e2c9cca29432049d453 P 237b3fcfab304c2c9dc95487d3968dce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:45:42.280751 9514 raft_consensus.cc:3037] T 220f79826f574e2c9cca29432049d453 P 237b3fcfab304c2c9dc95487d3968dce [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:42.281867 9514 raft_consensus.cc:504] T 220f79826f574e2c9cca29432049d453 P 237b3fcfab304c2c9dc95487d3968dce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaa347195e04db2a66afc180fd6de79" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37775 } } peers { permanent_uuid: "edf2c469841842bebe56cab2c925b669" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37903 } } peers { permanent_uuid: "237b3fcfab304c2c9dc95487d3968dce" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 36885 } }
I20260806 02:45:42.282253 9514 leader_election.cc:288] T 220f79826f574e2c9cca29432049d453 P 237b3fcfab304c2c9dc95487d3968dce [CANDIDATE]: Term 1 election: Requested vote from peers 8eaa347195e04db2a66afc180fd6de79 (127.1.183.131:37775), edf2c469841842bebe56cab2c925b669 (127.1.183.130:37903)
I20260806 02:45:42.282625 9215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220f79826f574e2c9cca29432049d453" candidate_uuid: "237b3fcfab304c2c9dc95487d3968dce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edf2c469841842bebe56cab2c925b669"
I20260806 02:45:42.282634 9414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220f79826f574e2c9cca29432049d453" candidate_uuid: "237b3fcfab304c2c9dc95487d3968dce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eaa347195e04db2a66afc180fd6de79"
I20260806 02:45:42.282770 9215 raft_consensus.cc:3037] T 220f79826f574e2c9cca29432049d453 P edf2c469841842bebe56cab2c925b669 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:42.282784 9414 raft_consensus.cc:3037] T 220f79826f574e2c9cca29432049d453 P 8eaa347195e04db2a66afc180fd6de79 [term 0 FOLLOWER]: Advancing to term 1
W20260806 02:45:42.283272 9083 tablet.cc:2369] T 220f79826f574e2c9cca29432049d453 P 237b3fcfab304c2c9dc95487d3968dce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:45:42.283963 9414 raft_consensus.cc:2454] T 220f79826f574e2c9cca29432049d453 P 8eaa347195e04db2a66afc180fd6de79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 237b3fcfab304c2c9dc95487d3968dce in term 1.
I20260806 02:45:42.284201 9215 raft_consensus.cc:2454] T 220f79826f574e2c9cca29432049d453 P edf2c469841842bebe56cab2c925b669 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 237b3fcfab304c2c9dc95487d3968dce in term 1.
I20260806 02:45:42.284354 8922 leader_election.cc:302] T 220f79826f574e2c9cca29432049d453 P 237b3fcfab304c2c9dc95487d3968dce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 237b3fcfab304c2c9dc95487d3968dce, 8eaa347195e04db2a66afc180fd6de79; no voters:
I20260806 02:45:42.284533 9514 raft_consensus.cc:2781] T 220f79826f574e2c9cca29432049d453 P 237b3fcfab304c2c9dc95487d3968dce [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:42.284787 9514 raft_consensus.cc:686] T 220f79826f574e2c9cca29432049d453 P 237b3fcfab304c2c9dc95487d3968dce [term 1 LEADER]: Becoming Leader. State: Replica: 237b3fcfab304c2c9dc95487d3968dce, State: Running, Role: LEADER
I20260806 02:45:42.285193 9514 consensus_queue.cc:237] T 220f79826f574e2c9cca29432049d453 P 237b3fcfab304c2c9dc95487d3968dce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaa347195e04db2a66afc180fd6de79" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37775 } } peers { permanent_uuid: "edf2c469841842bebe56cab2c925b669" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37903 } } peers { permanent_uuid: "237b3fcfab304c2c9dc95487d3968dce" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 36885 } }
I20260806 02:45:42.288698 8821 catalog_manager.cc:5495] T 220f79826f574e2c9cca29432049d453 P 237b3fcfab304c2c9dc95487d3968dce reported cstate change: term changed from 0 to 1, leader changed from <none> to 237b3fcfab304c2c9dc95487d3968dce (127.1.183.129). New cstate: current_term: 1 leader_uuid: "237b3fcfab304c2c9dc95487d3968dce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaa347195e04db2a66afc180fd6de79" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edf2c469841842bebe56cab2c925b669" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "237b3fcfab304c2c9dc95487d3968dce" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 36885 } health_report { overall_health: HEALTHY } } }
I20260806 02:45:42.313431 9504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:42.311786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33854 (local address 127.1.183.129:36885)
0806 02:45:42.311935 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:42.311939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:42.311958 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:45:42.312143 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:42.312145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:42.312235 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:42.312310 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:42.313131 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:42.313133 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:42.313135 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:42.313143 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:42.313157 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:42.313164 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:42.313233 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:42.313277 (+ 44us) server_negotiation.cc:299] Negotiation successful
0806 02:45:42.313311 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:45:42.319911 9414 raft_consensus.cc:1261] T 220f79826f574e2c9cca29432049d453 P 8eaa347195e04db2a66afc180fd6de79 [term 1 FOLLOWER]: Refusing update from remote peer 237b3fcfab304c2c9dc95487d3968dce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:42.319911 9215 raft_consensus.cc:1261] T 220f79826f574e2c9cca29432049d453 P edf2c469841842bebe56cab2c925b669 [term 1 FOLLOWER]: Refusing update from remote peer 237b3fcfab304c2c9dc95487d3968dce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:42.320781 9514 consensus_queue.cc:1035] T 220f79826f574e2c9cca29432049d453 P 237b3fcfab304c2c9dc95487d3968dce [LEADER]: Connected to new peer: Peer: permanent_uuid: "edf2c469841842bebe56cab2c925b669" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:42.321058 9522 consensus_queue.cc:1035] T 220f79826f574e2c9cca29432049d453 P 237b3fcfab304c2c9dc95487d3968dce [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eaa347195e04db2a66afc180fd6de79" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:42.337487 8818 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:42.337649 8818 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:42.340245 8818 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2319e2c1ca8406e9040bc22f9577d55: Sending DeleteTablet for 3 replicas of tablet 220f79826f574e2c9cca29432049d453
I20260806 02:45:42.341164 9381 tablet_service.cc:1514] Processing DeleteTablet for tablet 220f79826f574e2c9cca29432049d453 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:42 UTC) from {username='slave'} at 127.0.0.1:38464
I20260806 02:45:42.341308 9181 tablet_service.cc:1514] Processing DeleteTablet for tablet 220f79826f574e2c9cca29432049d453 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:42 UTC) from {username='slave'} at 127.0.0.1:47202
I20260806 02:45:42.341308 8989 tablet_service.cc:1514] Processing DeleteTablet for tablet 220f79826f574e2c9cca29432049d453 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:42 UTC) from {username='slave'} at 127.0.0.1:46962
I20260806 02:45:42.341982 9541 tablet_replica.cc:330] T 220f79826f574e2c9cca29432049d453 P 8eaa347195e04db2a66afc180fd6de79: stopping tablet replica
I20260806 02:45:42.342013 9543 tablet_replica.cc:330] T 220f79826f574e2c9cca29432049d453 P 237b3fcfab304c2c9dc95487d3968dce: stopping tablet replica
I20260806 02:45:42.342332 9543 raft_consensus.cc:2229] T 220f79826f574e2c9cca29432049d453 P 237b3fcfab304c2c9dc95487d3968dce [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:45:42.342360 9541 raft_consensus.cc:2229] T 220f79826f574e2c9cca29432049d453 P 8eaa347195e04db2a66afc180fd6de79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:42.342667 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 8902
I20260806 02:45:42.342751 9541 raft_consensus.cc:2258] T 220f79826f574e2c9cca29432049d453 P 8eaa347195e04db2a66afc180fd6de79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:42.342769 9543 raft_consensus.cc:2258] T 220f79826f574e2c9cca29432049d453 P 237b3fcfab304c2c9dc95487d3968dce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:42.343169 9542 tablet_replica.cc:330] T 220f79826f574e2c9cca29432049d453 P edf2c469841842bebe56cab2c925b669: stopping tablet replica
I20260806 02:45:42.343528 9542 raft_consensus.cc:2229] T 220f79826f574e2c9cca29432049d453 P edf2c469841842bebe56cab2c925b669 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:42.343847 9542 raft_consensus.cc:2258] T 220f79826f574e2c9cca29432049d453 P edf2c469841842bebe56cab2c925b669 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:42.347003 9542 ts_tablet_manager.cc:1905] T 220f79826f574e2c9cca29432049d453 P edf2c469841842bebe56cab2c925b669: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:42.347136 9541 ts_tablet_manager.cc:1905] T 220f79826f574e2c9cca29432049d453 P 8eaa347195e04db2a66afc180fd6de79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:42.349624 9542 ts_tablet_manager.cc:1918] T 220f79826f574e2c9cca29432049d453 P edf2c469841842bebe56cab2c925b669: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:45:42.349759 9542 log.cc:1196] T 220f79826f574e2c9cca29432049d453 P edf2c469841842bebe56cab2c925b669: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-1/wal/wals/220f79826f574e2c9cca29432049d453
I20260806 02:45:42.350062 9542 ts_tablet_manager.cc:1939] T 220f79826f574e2c9cca29432049d453 P edf2c469841842bebe56cab2c925b669: Deleting consensus metadata
I20260806 02:45:42.351173 9541 ts_tablet_manager.cc:1918] T 220f79826f574e2c9cca29432049d453 P 8eaa347195e04db2a66afc180fd6de79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:45:42.351393 9541 log.cc:1196] T 220f79826f574e2c9cca29432049d453 P 8eaa347195e04db2a66afc180fd6de79: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984313887056-1758-0/minicluster-data/ts-2/wal/wals/220f79826f574e2c9cca29432049d453
I20260806 02:45:42.351405 8801 catalog_manager.cc:4841] TS edf2c469841842bebe56cab2c925b669 (127.1.183.130:37903): tablet 220f79826f574e2c9cca29432049d453 (table test-table [id=4234a1d8530943e7b792ff7f37c0aad0]) successfully deleted
I20260806 02:45:42.351871 9541 ts_tablet_manager.cc:1939] T 220f79826f574e2c9cca29432049d453 P 8eaa347195e04db2a66afc180fd6de79: Deleting consensus metadata
I20260806 02:45:42.353655 8793 catalog_manager.cc:4841] TS 8eaa347195e04db2a66afc180fd6de79 (127.1.183.131:37775): tablet 220f79826f574e2c9cca29432049d453 (table test-table [id=4234a1d8530943e7b792ff7f37c0aad0]) successfully deleted
W20260806 02:45:42.359103 8793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.183.129:36885 (error 108)
I20260806 02:45:42.359365 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 9086
I20260806 02:45:42.360966 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:42.360468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:36885 (local address 127.0.0.1:33870)
0806 02:45:42.360730 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:42.360843 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.129:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260806 02:45:42.361192 8793 catalog_manager.cc:4584] TS 237b3fcfab304c2c9dc95487d3968dce (127.1.183.129:36885): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 220f79826f574e2c9cca29432049d453: Network error: Client connection negotiation failed: client connection to 127.1.183.129:36885: connect: Connection refused (error 111)
I20260806 02:45:42.373001 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 9282
I20260806 02:45:42.386680 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 8770
2026-08-06T02:45:42Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1071 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260806 02:45:42.420111 1758 mini_oidc.cc:152] Starting JWKS server
I20260806 02:45:42.422034 1758 webserver.cc:460] Webserver started at https://127.0.0.1:35519/ using document root <none> and password file <none>
I20260806 02:45:42.422197 1758 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 02:45:42.422462 1758 webserver.cc:460] Webserver started at http://127.0.0.1:36451/ using document root <none> and password file <none>
2026-08-06T02:45:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:42Z Disabled control of system clock
I20260806 02:45:42.433748 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:38581
--webserver_interface=127.1.183.190
--webserver_port=0
--builtin_ntp_servers=127.1.183.148:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:38581
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35519/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984313887056-1758-0/testchainca.cert with env {}
W20260806 02:45:42.555150 9552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:42.555472 9552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:42.555516 9552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:42.559707 9552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:42.559854 9552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:42.559883 9552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:42.559918 9552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:42.559947 9552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 02:45:42.559978 9552 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 02:45:42.559993 9552 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35519/jwks.json
I20260806 02:45:42.565797 9552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:38581
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:38581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35519/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984313887056-1758-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:42.566946 9552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:42.567979 9552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:42.577121 9560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:42.577194 9563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:42.577337 9552 server_base.cc:1034] running on GCE node
W20260806 02:45:42.577121 9561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:42.578295 9552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:42.579497 9552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:42.580663 9552 hybrid_clock.cc:648] HybridClock initialized: now 1785984342580664 us; error 45 us; skew 500 ppm
I20260806 02:45:42.582760 9552 webserver.cc:460] Webserver started at http://127.1.183.190:34847/ using document root <none> and password file <none>
I20260806 02:45:42.583403 9552 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:42.583472 9552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:42.583704 9552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:42.585963 9552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "e4b0a47fbf1a4ec6ab98a05df25ecbbb"
format_stamp: "Formatted at 2026-08-06 02:45:42 on dist-test-slave-gmjp"
server_key: "ceee9da6260e36e75e674c3bace60eb8"
server_key_iv: "54fe3dfa66e2285105fe7118cbcb35a6"
server_key_version: "encryptionkey@0"
I20260806 02:45:42.586602 9552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "e4b0a47fbf1a4ec6ab98a05df25ecbbb"
format_stamp: "Formatted at 2026-08-06 02:45:42 on dist-test-slave-gmjp"
server_key: "ceee9da6260e36e75e674c3bace60eb8"
server_key_iv: "54fe3dfa66e2285105fe7118cbcb35a6"
server_key_version: "encryptionkey@0"
I20260806 02:45:42.590700 9552 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260806 02:45:42.593380 9570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:42.594749 9552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:42.594887 9552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "e4b0a47fbf1a4ec6ab98a05df25ecbbb"
format_stamp: "Formatted at 2026-08-06 02:45:42 on dist-test-slave-gmjp"
server_key: "ceee9da6260e36e75e674c3bace60eb8"
server_key_iv: "54fe3dfa66e2285105fe7118cbcb35a6"
server_key_version: "encryptionkey@0"
I20260806 02:45:42.594986 9552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:42.625540 9552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:42.626255 9552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:42.626416 9552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:42.634209 9552 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:38581
I20260806 02:45:42.634253 9645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:38581 every 8 connection(s)
I20260806 02:45:42.635252 9552 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:42.638882 9647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:42.641553 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 9552
I20260806 02:45:42.641705 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:45:42.642067 1758 external_mini_cluster.cc:1428] Setting key e4c4b78c0c241ccd744d661186cc2492
I20260806 02:45:42.645901 9647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4b0a47fbf1a4ec6ab98a05df25ecbbb: Bootstrap starting.
I20260806 02:45:42.648801 9647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4b0a47fbf1a4ec6ab98a05df25ecbbb: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:42.649766 9647 log.cc:824] T 00000000000000000000000000000000 P e4b0a47fbf1a4ec6ab98a05df25ecbbb: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:42.652546 9647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4b0a47fbf1a4ec6ab98a05df25ecbbb: No bootstrap required, opened a new log
I20260806 02:45:42.652876 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:42.643812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50846 (local address 127.1.183.190:38581)
0806 02:45:42.644590 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:42.644606 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:42.644643 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:45:42.645334 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:42.645352 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:42.645789 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:42.646165 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:42.646552 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:42.647574 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:42.648973 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:42.649484 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:42.649606 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:42.650579 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:42.650585 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:42.650605 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:42.651113 (+ 508us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:42.651171 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:42.651193 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:42.651275 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:42.651573 (+ 298us) server_negotiation.cc:299] Negotiation successful
0806 02:45:42.651950 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":149,"threads_started":1}
I20260806 02:45:42.655918 1758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:45:42.657121 9647 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4b0a47fbf1a4ec6ab98a05df25ecbbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b0a47fbf1a4ec6ab98a05df25ecbbb" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38581 } }
I20260806 02:45:42.657379 9647 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4b0a47fbf1a4ec6ab98a05df25ecbbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:42.657462 9647 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4b0a47fbf1a4ec6ab98a05df25ecbbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4b0a47fbf1a4ec6ab98a05df25ecbbb, State: Initialized, Role: FOLLOWER
I20260806 02:45:42.658011 9647 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4b0a47fbf1a4ec6ab98a05df25ecbbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b0a47fbf1a4ec6ab98a05df25ecbbb" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38581 } }
I20260806 02:45:42.658166 9647 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4b0a47fbf1a4ec6ab98a05df25ecbbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:42.658282 9647 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4b0a47fbf1a4ec6ab98a05df25ecbbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:42.658432 9647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4b0a47fbf1a4ec6ab98a05df25ecbbb [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:42.659561 9647 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4b0a47fbf1a4ec6ab98a05df25ecbbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b0a47fbf1a4ec6ab98a05df25ecbbb" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38581 } }
I20260806 02:45:42.660000 9647 leader_election.cc:302] T 00000000000000000000000000000000 P e4b0a47fbf1a4ec6ab98a05df25ecbbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4b0a47fbf1a4ec6ab98a05df25ecbbb; no voters:
I20260806 02:45:42.660456 9647 leader_election.cc:288] T 00000000000000000000000000000000 P e4b0a47fbf1a4ec6ab98a05df25ecbbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:42.660610 9654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4b0a47fbf1a4ec6ab98a05df25ecbbb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:42.660902 9654 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4b0a47fbf1a4ec6ab98a05df25ecbbb [term 1 LEADER]: Becoming Leader. State: Replica: e4b0a47fbf1a4ec6ab98a05df25ecbbb, State: Running, Role: LEADER
I20260806 02:45:42.661242 9654 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4b0a47fbf1a4ec6ab98a05df25ecbbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b0a47fbf1a4ec6ab98a05df25ecbbb" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38581 } }
I20260806 02:45:42.662066 9647 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4b0a47fbf1a4ec6ab98a05df25ecbbb [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:42.663184 9655 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4b0a47fbf1a4ec6ab98a05df25ecbbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4b0a47fbf1a4ec6ab98a05df25ecbbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b0a47fbf1a4ec6ab98a05df25ecbbb" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38581 } } }
I20260806 02:45:42.663193 9656 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4b0a47fbf1a4ec6ab98a05df25ecbbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4b0a47fbf1a4ec6ab98a05df25ecbbb. Latest consensus state: current_term: 1 leader_uuid: "e4b0a47fbf1a4ec6ab98a05df25ecbbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b0a47fbf1a4ec6ab98a05df25ecbbb" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38581 } } }
I20260806 02:45:42.663369 9656 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4b0a47fbf1a4ec6ab98a05df25ecbbb [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:42.663369 9655 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4b0a47fbf1a4ec6ab98a05df25ecbbb [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:42.663791 9664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:42.666688 9664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:42.673451 9664 catalog_manager.cc:1347] Generated new cluster ID: b2b436e75c6049bc90cb35e665cfcc3b
I20260806 02:45:42.673532 9664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:42.680195 9664 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:42.681650 9664 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:42.687981 9664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4b0a47fbf1a4ec6ab98a05df25ecbbb: Generated new TSK 0
I20260806 02:45:42.688941 9664 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 02:45:42.735296 9683 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260806 02:45:42.735445 9683 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 02:45:42.735948 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:42.708891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50856 (local address 127.1.183.190:38581)
0806 02:45:42.709040 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:42.709047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:42.709297 (+ 250us) server_negotiation.cc:407] Connection header received
0806 02:45:42.709579 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:42.709581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:42.709638 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:42.709753 (+ 115us) server_negotiation.cc:226] Negotiated authn=JWT
0806 02:45:42.710052 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:42.710663 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:42.711593 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:42.711855 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:42.711972 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:42.712444 (+ 472us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 02:45:42.735445 (+ 23001us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35519/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 02:45:42.735791 (+ 346us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50856: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35519/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":56}
W20260806 02:45:42.736069 9651 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50856: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35519/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260806 02:45:42.737516 9682 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:42.708760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:38581 (local address 127.0.0.1:50856)
0806 02:45:42.709186 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:42.709199 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:42.709377 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:42.709774 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:42.709777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:42.709786 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0806 02:45:42.709924 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:42.709934 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:42.710798 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:42.710802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:42.711454 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:42.711462 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:42.711984 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:42.711988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:42.712089 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:42.712145 (+ 56us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 02:45:42.735770 (+ 23625us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35519/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 02:45:42.736063 (+ 293us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.183.190:38581: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35519/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":139,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.1.183.190:38581: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35519/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260806 02:45:42.738468 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 9552
2026-08-06T02:45:42Z chronyd exiting
I20260806 02:45:42.786562 1758 test_util.cc:175] -----------------------------------------------
I20260806 02:45:42.786661 1758 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984313887056-1758-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (368 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260806 02:45:42.787768 1758 mini_oidc.cc:152] Starting JWKS server
I20260806 02:45:42.788787 1758 webserver.cc:460] Webserver started at https://127.0.0.1:45545/ using document root <none> and password file <none>
I20260806 02:45:42.788877 1758 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 02:45:42.789155 1758 webserver.cc:460] Webserver started at http://127.0.0.1:43783/ using document root <none> and password file <none>
2026-08-06T02:45:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:42Z Disabled control of system clock
I20260806 02:45:42.800266 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:33713
--webserver_interface=127.1.183.190
--webserver_port=0
--builtin_ntp_servers=127.1.183.148:44943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:33713
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45545/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984313887056-1758-0/testchainca.cert with env {}
W20260806 02:45:42.918571 9691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:42.918898 9691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:42.918941 9691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:42.923158 9691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:42.923242 9691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:42.923259 9691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:42.923275 9691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:42.923291 9691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 02:45:42.923307 9691 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 02:45:42.923321 9691 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45545/jwks.json
I20260806 02:45:42.929179 9691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:44943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:33713
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:33713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45545/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984313887056-1758-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:42.930348 9691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:42.931396 9691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:42.940479 9700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:42.940479 9697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:42.940487 9698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:42.940783 9691 server_base.cc:1034] running on GCE node
I20260806 02:45:42.941520 9691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:42.942735 9691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:42.943909 9691 hybrid_clock.cc:648] HybridClock initialized: now 1785984342943896 us; error 31 us; skew 500 ppm
I20260806 02:45:42.945996 9691 webserver.cc:460] Webserver started at http://127.1.183.190:41993/ using document root <none> and password file <none>
I20260806 02:45:42.946642 9691 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:42.946707 9691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:42.946959 9691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:42.949257 9691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "badfaf2227ab4aea8d3cb53cb85ecde0"
format_stamp: "Formatted at 2026-08-06 02:45:42 on dist-test-slave-gmjp"
server_key: "459004c44b17910ce6ee82e988832bd4"
server_key_iv: "c78e29c6bb3da0875d5bc41a6a30b67b"
server_key_version: "encryptionkey@0"
I20260806 02:45:42.949898 9691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "badfaf2227ab4aea8d3cb53cb85ecde0"
format_stamp: "Formatted at 2026-08-06 02:45:42 on dist-test-slave-gmjp"
server_key: "459004c44b17910ce6ee82e988832bd4"
server_key_iv: "c78e29c6bb3da0875d5bc41a6a30b67b"
server_key_version: "encryptionkey@0"
I20260806 02:45:42.953874 9691 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 02:45:42.956796 9713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:42.958235 9691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:42.958398 9691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "badfaf2227ab4aea8d3cb53cb85ecde0"
format_stamp: "Formatted at 2026-08-06 02:45:42 on dist-test-slave-gmjp"
server_key: "459004c44b17910ce6ee82e988832bd4"
server_key_iv: "c78e29c6bb3da0875d5bc41a6a30b67b"
server_key_version: "encryptionkey@0"
I20260806 02:45:42.958542 9691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:42.975574 9691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:42.976498 9691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:42.976724 9691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:42.984330 9779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:33713 every 8 connection(s)
I20260806 02:45:42.984308 9691 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:33713
I20260806 02:45:42.985513 9691 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:42.987280 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 9691
I20260806 02:45:42.987421 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:45:42.987789 1758 external_mini_cluster.cc:1428] Setting key 6fba2eee613dbb26ccc4a8c3a2a901fe
I20260806 02:45:42.989702 9781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:42.995996 9781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P badfaf2227ab4aea8d3cb53cb85ecde0: Bootstrap starting.
I20260806 02:45:42.997812 9790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:42.989348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33738 (local address 127.1.183.190:33713)
0806 02:45:42.990128 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:42.990147 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:42.990186 (+ 39us) server_negotiation.cc:407] Connection header received
0806 02:45:42.990873 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:42.990890 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:42.991299 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:42.991663 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:42.992129 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:42.993193 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:42.994207 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:42.994707 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:42.994892 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:42.995864 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:42.995869 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:42.995883 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:42.996411 (+ 528us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:42.996473 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:42.996496 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:42.996612 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:42.996776 (+ 164us) server_negotiation.cc:299] Negotiation successful
0806 02:45:42.997055 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":165,"threads_started":1}
I20260806 02:45:42.998602 9781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P badfaf2227ab4aea8d3cb53cb85ecde0: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:42.999400 9781 log.cc:824] T 00000000000000000000000000000000 P badfaf2227ab4aea8d3cb53cb85ecde0: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:43.000914 1758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:45:43.001860 9781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P badfaf2227ab4aea8d3cb53cb85ecde0: No bootstrap required, opened a new log
I20260806 02:45:43.005962 9781 raft_consensus.cc:348] T 00000000000000000000000000000000 P badfaf2227ab4aea8d3cb53cb85ecde0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "badfaf2227ab4aea8d3cb53cb85ecde0" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33713 } }
I20260806 02:45:43.006294 9781 raft_consensus.cc:374] T 00000000000000000000000000000000 P badfaf2227ab4aea8d3cb53cb85ecde0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:43.006363 9781 raft_consensus.cc:729] T 00000000000000000000000000000000 P badfaf2227ab4aea8d3cb53cb85ecde0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: badfaf2227ab4aea8d3cb53cb85ecde0, State: Initialized, Role: FOLLOWER
I20260806 02:45:43.006910 9781 consensus_queue.cc:260] T 00000000000000000000000000000000 P badfaf2227ab4aea8d3cb53cb85ecde0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "badfaf2227ab4aea8d3cb53cb85ecde0" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33713 } }
I20260806 02:45:43.007066 9781 raft_consensus.cc:388] T 00000000000000000000000000000000 P badfaf2227ab4aea8d3cb53cb85ecde0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:43.007117 9781 raft_consensus.cc:482] T 00000000000000000000000000000000 P badfaf2227ab4aea8d3cb53cb85ecde0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:43.007192 9781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P badfaf2227ab4aea8d3cb53cb85ecde0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:43.008215 9781 raft_consensus.cc:504] T 00000000000000000000000000000000 P badfaf2227ab4aea8d3cb53cb85ecde0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "badfaf2227ab4aea8d3cb53cb85ecde0" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33713 } }
I20260806 02:45:43.008550 9781 leader_election.cc:302] T 00000000000000000000000000000000 P badfaf2227ab4aea8d3cb53cb85ecde0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: badfaf2227ab4aea8d3cb53cb85ecde0; no voters:
I20260806 02:45:43.008836 9781 leader_election.cc:288] T 00000000000000000000000000000000 P badfaf2227ab4aea8d3cb53cb85ecde0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:43.009001 9792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P badfaf2227ab4aea8d3cb53cb85ecde0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:43.009253 9792 raft_consensus.cc:686] T 00000000000000000000000000000000 P badfaf2227ab4aea8d3cb53cb85ecde0 [term 1 LEADER]: Becoming Leader. State: Replica: badfaf2227ab4aea8d3cb53cb85ecde0, State: Running, Role: LEADER
I20260806 02:45:43.009589 9792 consensus_queue.cc:237] T 00000000000000000000000000000000 P badfaf2227ab4aea8d3cb53cb85ecde0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "badfaf2227ab4aea8d3cb53cb85ecde0" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33713 } }
I20260806 02:45:43.010176 9781 sys_catalog.cc:564] T 00000000000000000000000000000000 P badfaf2227ab4aea8d3cb53cb85ecde0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:43.011372 9794 sys_catalog.cc:455] T 00000000000000000000000000000000 P badfaf2227ab4aea8d3cb53cb85ecde0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader badfaf2227ab4aea8d3cb53cb85ecde0. Latest consensus state: current_term: 1 leader_uuid: "badfaf2227ab4aea8d3cb53cb85ecde0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "badfaf2227ab4aea8d3cb53cb85ecde0" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33713 } } }
I20260806 02:45:43.011510 9794 sys_catalog.cc:458] T 00000000000000000000000000000000 P badfaf2227ab4aea8d3cb53cb85ecde0 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:43.011811 9793 sys_catalog.cc:455] T 00000000000000000000000000000000 P badfaf2227ab4aea8d3cb53cb85ecde0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "badfaf2227ab4aea8d3cb53cb85ecde0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "badfaf2227ab4aea8d3cb53cb85ecde0" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33713 } } }
I20260806 02:45:43.011973 9793 sys_catalog.cc:458] T 00000000000000000000000000000000 P badfaf2227ab4aea8d3cb53cb85ecde0 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:43.012006 9806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:43.015338 9806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:43.021512 9806 catalog_manager.cc:1347] Generated new cluster ID: b1164e48d9654e7eab7d8feeaf7537e4
I20260806 02:45:43.021596 9806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:43.036257 9806 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:43.037352 9806 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:43.048243 9806 catalog_manager.cc:5868] T 00000000000000000000000000000000 P badfaf2227ab4aea8d3cb53cb85ecde0: Generated new TSK 0
I20260806 02:45:43.049053 9806 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 02:45:43.080653 9825 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260806 02:45:43.080765 9825 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 02:45:43.081624 9790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:43.064978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33752 (local address 127.1.183.190:33713)
0806 02:45:43.065140 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:43.065145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:43.065296 (+ 151us) server_negotiation.cc:407] Connection header received
0806 02:45:43.065474 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:43.065478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:43.065540 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:43.065665 (+ 125us) server_negotiation.cc:226] Negotiated authn=JWT
0806 02:45:43.065877 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:43.066723 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:43.067615 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:43.067972 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:43.068085 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:43.068485 (+ 400us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 02:45:43.081114 (+ 12629us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45545/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 02:45:43.081467 (+ 353us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33752: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45545/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":60}
W20260806 02:45:43.081712 9790 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33752: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45545/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260806 02:45:43.081765 9824 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:43.064904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:33713 (local address 127.0.0.1:33752)
0806 02:45:43.065194 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:43.065210 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:43.065324 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:43.065668 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:43.065670 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:43.065676 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0806 02:45:43.065773 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:43.065777 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:43.066842 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:43.066844 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:43.067491 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:43.067499 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:43.068105 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:43.068108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:43.068244 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:43.068271 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 02:45:43.081392 (+ 13121us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45545/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 02:45:43.081567 (+ 175us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.183.190:33713: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45545/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":113,"threads_started":1}
I20260806 02:45:43.082472 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 9691
2026-08-06T02:45:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (345 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260806 02:45:43.133625 1758 mini_oidc.cc:152] Starting JWKS server
I20260806 02:45:43.134552 1758 webserver.cc:460] Webserver started at https://127.0.0.1:41755/ using document root <none> and password file <none>
I20260806 02:45:43.134639 1758 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 02:45:43.134930 1758 webserver.cc:460] Webserver started at http://127.0.0.1:45621/ using document root <none> and password file <none>
2026-08-06T02:45:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:43Z Disabled control of system clock
I20260806 02:45:43.146013 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:45293
--webserver_interface=127.1.183.190
--webserver_port=0
--builtin_ntp_servers=127.1.183.148:36299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:45293
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41755/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 02:45:43.266458 9838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:43.266834 9838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:43.266928 9838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:43.271247 9838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:43.271368 9838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:43.271394 9838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:43.271425 9838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:43.271462 9838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 02:45:43.271492 9838 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 02:45:43.271517 9838 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41755/jwks.json
I20260806 02:45:43.276687 9838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:36299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:45293
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:45293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41755/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:43.277961 9838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:43.279531 9838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:43.288806 9848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:43.288826 9844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:43.288923 9845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:43.289063 9838 server_base.cc:1034] running on GCE node
I20260806 02:45:43.289736 9838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:43.290962 9838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:43.292232 9838 hybrid_clock.cc:648] HybridClock initialized: now 1785984343292206 us; error 33 us; skew 500 ppm
I20260806 02:45:43.294346 9838 webserver.cc:460] Webserver started at http://127.1.183.190:32985/ using document root <none> and password file <none>
I20260806 02:45:43.294992 9838 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:43.295095 9838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:43.295363 9838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:43.297788 9838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "b2ec6d1958c941ad8333bf818a1cbe4e"
format_stamp: "Formatted at 2026-08-06 02:45:43 on dist-test-slave-gmjp"
server_key: "e8ab0b07fd7680df6c9ff7c96985cfaf"
server_key_iv: "d8f3ebdcd6a08fa6d7e23066c530b634"
server_key_version: "encryptionkey@0"
I20260806 02:45:43.298476 9838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "b2ec6d1958c941ad8333bf818a1cbe4e"
format_stamp: "Formatted at 2026-08-06 02:45:43 on dist-test-slave-gmjp"
server_key: "e8ab0b07fd7680df6c9ff7c96985cfaf"
server_key_iv: "d8f3ebdcd6a08fa6d7e23066c530b634"
server_key_version: "encryptionkey@0"
I20260806 02:45:43.302790 9838 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:45:43.305755 9856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:43.307125 9838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:45:43.307308 9838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "b2ec6d1958c941ad8333bf818a1cbe4e"
format_stamp: "Formatted at 2026-08-06 02:45:43 on dist-test-slave-gmjp"
server_key: "e8ab0b07fd7680df6c9ff7c96985cfaf"
server_key_iv: "d8f3ebdcd6a08fa6d7e23066c530b634"
server_key_version: "encryptionkey@0"
I20260806 02:45:43.307459 9838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:43.319300 9838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:43.320089 9838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:43.320338 9838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:43.327695 9838 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:45293
I20260806 02:45:43.327715 9928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:45293 every 8 connection(s)
I20260806 02:45:43.328745 9838 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:43.332104 9929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:43.332834 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 9838
I20260806 02:45:43.332940 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:45:43.333262 1758 external_mini_cluster.cc:1428] Setting key c281212dd75caaf546b5dde343afe585
I20260806 02:45:43.339001 9929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2ec6d1958c941ad8333bf818a1cbe4e: Bootstrap starting.
I20260806 02:45:43.342031 9929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2ec6d1958c941ad8333bf818a1cbe4e: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:43.342928 9929 log.cc:824] T 00000000000000000000000000000000 P b2ec6d1958c941ad8333bf818a1cbe4e: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:43.343638 9933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:43.334774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42368 (local address 127.1.183.190:45293)
0806 02:45:43.335450 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:43.335467 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:43.335506 (+ 39us) server_negotiation.cc:407] Connection header received
0806 02:45:43.336215 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:43.336241 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:43.336672 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:43.336972 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:43.337405 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:43.338408 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:43.339645 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:43.340405 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:43.340535 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:43.341462 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:43.341467 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:43.341481 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:43.341918 (+ 437us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:43.341978 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:43.341999 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:43.342079 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:43.342323 (+ 244us) server_negotiation.cc:299] Negotiation successful
0806 02:45:43.342652 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":155,"threads_started":1}
I20260806 02:45:43.345383 9929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2ec6d1958c941ad8333bf818a1cbe4e: No bootstrap required, opened a new log
I20260806 02:45:43.346869 1758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:45:43.349898 9929 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2ec6d1958c941ad8333bf818a1cbe4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ec6d1958c941ad8333bf818a1cbe4e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 45293 } }
I20260806 02:45:43.350239 9929 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2ec6d1958c941ad8333bf818a1cbe4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:43.350307 9929 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2ec6d1958c941ad8333bf818a1cbe4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2ec6d1958c941ad8333bf818a1cbe4e, State: Initialized, Role: FOLLOWER
I20260806 02:45:43.350746 9929 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2ec6d1958c941ad8333bf818a1cbe4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ec6d1958c941ad8333bf818a1cbe4e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 45293 } }
I20260806 02:45:43.350917 9929 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2ec6d1958c941ad8333bf818a1cbe4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:43.351011 9929 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2ec6d1958c941ad8333bf818a1cbe4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:43.351118 9929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2ec6d1958c941ad8333bf818a1cbe4e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:43.352434 9929 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2ec6d1958c941ad8333bf818a1cbe4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ec6d1958c941ad8333bf818a1cbe4e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 45293 } }
I20260806 02:45:43.352757 9929 leader_election.cc:302] T 00000000000000000000000000000000 P b2ec6d1958c941ad8333bf818a1cbe4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2ec6d1958c941ad8333bf818a1cbe4e; no voters:
I20260806 02:45:43.353057 9929 leader_election.cc:288] T 00000000000000000000000000000000 P b2ec6d1958c941ad8333bf818a1cbe4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:43.353202 9936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2ec6d1958c941ad8333bf818a1cbe4e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:43.353480 9936 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2ec6d1958c941ad8333bf818a1cbe4e [term 1 LEADER]: Becoming Leader. State: Replica: b2ec6d1958c941ad8333bf818a1cbe4e, State: Running, Role: LEADER
I20260806 02:45:43.353822 9936 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2ec6d1958c941ad8333bf818a1cbe4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ec6d1958c941ad8333bf818a1cbe4e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 45293 } }
I20260806 02:45:43.354462 9929 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2ec6d1958c941ad8333bf818a1cbe4e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:43.355824 9938 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2ec6d1958c941ad8333bf818a1cbe4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2ec6d1958c941ad8333bf818a1cbe4e. Latest consensus state: current_term: 1 leader_uuid: "b2ec6d1958c941ad8333bf818a1cbe4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ec6d1958c941ad8333bf818a1cbe4e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 45293 } } }
I20260806 02:45:43.355993 9938 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2ec6d1958c941ad8333bf818a1cbe4e [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:43.356244 9937 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2ec6d1958c941ad8333bf818a1cbe4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2ec6d1958c941ad8333bf818a1cbe4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ec6d1958c941ad8333bf818a1cbe4e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 45293 } } }
I20260806 02:45:43.356417 9937 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2ec6d1958c941ad8333bf818a1cbe4e [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:43.357026 9947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:43.360545 9947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:43.366156 9947 catalog_manager.cc:1347] Generated new cluster ID: deb157598b524ae0af4168976f23d008
I20260806 02:45:43.366259 9947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:43.390797 9947 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:43.391837 9947 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:43.402129 9947 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2ec6d1958c941ad8333bf818a1cbe4e: Generated new TSK 0
I20260806 02:45:43.402932 9947 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:45:43.420863 9933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:43.415366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42372 (local address 127.1.183.190:45293)
0806 02:45:43.415515 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:43.415521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:43.415682 (+ 161us) server_negotiation.cc:407] Connection header received
0806 02:45:43.415824 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:43.415828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:43.415891 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:43.416021 (+ 130us) server_negotiation.cc:226] Negotiated authn=JWT
0806 02:45:43.416274 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:43.417151 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:43.418040 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:43.418383 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:43.418506 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:43.418802 (+ 296us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 02:45:43.420462 (+ 1660us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41755/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 02:45:43.420712 (+ 250us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42372: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41755/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":56}
W20260806 02:45:43.420956 9933 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42372: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41755/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260806 02:45:43.422679 9970 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:43.415232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:45293 (local address 127.0.0.1:42372)
0806 02:45:43.415593 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:43.415607 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:43.415709 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:43.416029 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:43.416031 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:43.416038 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0806 02:45:43.416140 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:43.416145 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:43.417275 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:43.417278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:43.417921 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:43.417930 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:43.418528 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:43.418531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:43.418594 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:43.418619 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 02:45:43.420638 (+ 2019us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41755/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 02:45:43.422494 (+ 1856us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.183.190:45293: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41755/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":1603,"thread_start_us":114,"threads_started":1}
I20260806 02:45:43.423807 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 9838
E20260806 02:45:43.431890 9971 webserver.cc:533] Webserver: sslize failed:
E20260806 02:45:43.432008 9971 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-06T02:45:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (335 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-06T02:45:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:43Z Disabled control of system clock
I20260806 02:45:43.479584 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:33105
--webserver_interface=127.1.183.190
--webserver_port=0
--builtin_ntp_servers=127.1.183.148:36903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:33105
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984313887056-1758-0/testchainca.cert with env {}
W20260806 02:45:43.596508 9976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:43.596799 9976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:43.596843 9976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:43.601024 9976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:43.601094 9976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:43.601109 9976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:43.601127 9976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:43.601143 9976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 02:45:43.601159 9976 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 02:45:43.601172 9976 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260806 02:45:43.606861 9976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:36903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:33105
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:33105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984313887056-1758-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:43.607991 9976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:43.609054 9976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:43.618103 9986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:43.618275 9976 server_base.cc:1034] running on GCE node
W20260806 02:45:43.618124 9990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:43.618505 9987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:43.619174 9976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:43.620420 9976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:43.621604 9976 hybrid_clock.cc:648] HybridClock initialized: now 1785984343621570 us; error 48 us; skew 500 ppm
I20260806 02:45:43.623840 9976 webserver.cc:460] Webserver started at http://127.1.183.190:41753/ using document root <none> and password file <none>
I20260806 02:45:43.624547 9976 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:43.624619 9976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:43.624850 9976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:43.627156 9976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "87d27ff3ce9d4ac392b354709c3d49be"
format_stamp: "Formatted at 2026-08-06 02:45:43 on dist-test-slave-gmjp"
server_key: "70afd158d3d6f3d63b74eedd8f71aa33"
server_key_iv: "7bd9a85f6e0b4022470d9c54c2951729"
server_key_version: "encryptionkey@0"
I20260806 02:45:43.627780 9976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "87d27ff3ce9d4ac392b354709c3d49be"
format_stamp: "Formatted at 2026-08-06 02:45:43 on dist-test-slave-gmjp"
server_key: "70afd158d3d6f3d63b74eedd8f71aa33"
server_key_iv: "7bd9a85f6e0b4022470d9c54c2951729"
server_key_version: "encryptionkey@0"
I20260806 02:45:43.631721 9976 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:45:43.634485 9998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:43.635835 9976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:45:43.635998 9976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "87d27ff3ce9d4ac392b354709c3d49be"
format_stamp: "Formatted at 2026-08-06 02:45:43 on dist-test-slave-gmjp"
server_key: "70afd158d3d6f3d63b74eedd8f71aa33"
server_key_iv: "7bd9a85f6e0b4022470d9c54c2951729"
server_key_version: "encryptionkey@0"
I20260806 02:45:43.636132 9976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:43.654594 9976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:43.655424 9976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:43.655655 9976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:43.663360 9976 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:33105
I20260806 02:45:43.663368 10083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:33105 every 8 connection(s)
I20260806 02:45:43.664433 9976 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:43.666585 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 9976
I20260806 02:45:43.666716 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:45:43.667050 1758 external_mini_cluster.cc:1428] Setting key 5a85fb72f9fcd9fc115ec4f7a55b8019
I20260806 02:45:43.668232 10084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:43.675441 10084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87d27ff3ce9d4ac392b354709c3d49be: Bootstrap starting.
I20260806 02:45:43.678188 10091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:43.668623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40910 (local address 127.1.183.190:33105)
0806 02:45:43.669387 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:43.669409 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:43.669457 (+ 48us) server_negotiation.cc:407] Connection header received
0806 02:45:43.670347 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:43.670371 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:43.670898 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:43.671289 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:43.671757 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:43.673233 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:43.674244 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:43.674944 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:43.675108 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:43.676030 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:43.676036 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:43.676052 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:45:43.676592 (+ 540us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:43.676665 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:45:43.676699 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:43.676811 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:43.676958 (+ 147us) server_negotiation.cc:299] Negotiation successful
0806 02:45:43.677236 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":112,"threads_started":1}
I20260806 02:45:43.678365 10084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87d27ff3ce9d4ac392b354709c3d49be: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:43.679165 10084 log.cc:824] T 00000000000000000000000000000000 P 87d27ff3ce9d4ac392b354709c3d49be: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:43.681488 1758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:45:43.681572 10084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87d27ff3ce9d4ac392b354709c3d49be: No bootstrap required, opened a new log
I20260806 02:45:43.685527 10084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87d27ff3ce9d4ac392b354709c3d49be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d27ff3ce9d4ac392b354709c3d49be" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33105 } }
I20260806 02:45:43.685758 10084 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87d27ff3ce9d4ac392b354709c3d49be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:43.685793 10084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87d27ff3ce9d4ac392b354709c3d49be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87d27ff3ce9d4ac392b354709c3d49be, State: Initialized, Role: FOLLOWER
I20260806 02:45:43.686165 10084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87d27ff3ce9d4ac392b354709c3d49be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d27ff3ce9d4ac392b354709c3d49be" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33105 } }
I20260806 02:45:43.686276 10084 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87d27ff3ce9d4ac392b354709c3d49be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:43.686316 10084 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87d27ff3ce9d4ac392b354709c3d49be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:43.686381 10084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87d27ff3ce9d4ac392b354709c3d49be [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:43.687279 10084 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87d27ff3ce9d4ac392b354709c3d49be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d27ff3ce9d4ac392b354709c3d49be" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33105 } }
I20260806 02:45:43.687626 10084 leader_election.cc:302] T 00000000000000000000000000000000 P 87d27ff3ce9d4ac392b354709c3d49be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87d27ff3ce9d4ac392b354709c3d49be; no voters:
I20260806 02:45:43.687908 10084 leader_election.cc:288] T 00000000000000000000000000000000 P 87d27ff3ce9d4ac392b354709c3d49be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:43.688071 10097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87d27ff3ce9d4ac392b354709c3d49be [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:43.688345 10097 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87d27ff3ce9d4ac392b354709c3d49be [term 1 LEADER]: Becoming Leader. State: Replica: 87d27ff3ce9d4ac392b354709c3d49be, State: Running, Role: LEADER
I20260806 02:45:43.688659 10097 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87d27ff3ce9d4ac392b354709c3d49be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d27ff3ce9d4ac392b354709c3d49be" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33105 } }
I20260806 02:45:43.689325 10084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87d27ff3ce9d4ac392b354709c3d49be [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:43.690663 10100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87d27ff3ce9d4ac392b354709c3d49be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87d27ff3ce9d4ac392b354709c3d49be. Latest consensus state: current_term: 1 leader_uuid: "87d27ff3ce9d4ac392b354709c3d49be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d27ff3ce9d4ac392b354709c3d49be" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33105 } } }
I20260806 02:45:43.690696 10099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87d27ff3ce9d4ac392b354709c3d49be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87d27ff3ce9d4ac392b354709c3d49be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d27ff3ce9d4ac392b354709c3d49be" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33105 } } }
I20260806 02:45:43.690798 10100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87d27ff3ce9d4ac392b354709c3d49be [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:43.690798 10099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87d27ff3ce9d4ac392b354709c3d49be [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:43.692095 10112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:43.695432 10112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:43.701399 10112 catalog_manager.cc:1347] Generated new cluster ID: 7c21a765f83549a1993f18363a42d958
I20260806 02:45:43.701483 10112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:43.710172 10112 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:43.711519 10112 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:43.721334 10112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87d27ff3ce9d4ac392b354709c3d49be: Generated new TSK 0
I20260806 02:45:43.722232 10112 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:45:44.179733 10091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:43.733077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40922 (local address 127.1.183.190:33105)
0806 02:45:43.733209 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:43.733213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:43.733398 (+ 185us) server_negotiation.cc:407] Connection header received
0806 02:45:43.733607 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:43.733610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:43.733658 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:43.733777 (+ 119us) server_negotiation.cc:226] Negotiated authn=JWT
0806 02:45:43.734031 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:43.734577 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:43.735498 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:43.735713 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:43.735830 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:43.736204 (+ 374us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 02:45:44.179180 (+442976us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 02:45:44.179560 (+ 380us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40922: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":46}
W20260806 02:45:44.179847 10091 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40922: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260806 02:45:44.179862 10135 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:43.732983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:33105 (local address 127.0.0.1:40922)
0806 02:45:43.733295 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:43.733317 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:43.733453 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:43.733777 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:43.733780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:43.733789 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0806 02:45:43.733910 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:43.733916 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:43.734699 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:43.734703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:43.735368 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:43.735376 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:43.735846 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:43.735850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:43.735941 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:43.735974 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 02:45:44.179454 (+443480us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 02:45:44.179667 (+ 213us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.183.190:33105: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":107,"threads_started":1}
I20260806 02:45:44.180779 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 9976
2026-08-06T02:45:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (739 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (132 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (138 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:45:44 dist-test-slave-gmjp krb5kdc[10152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:45:44 dist-test-slave-gmjp krb5kdc[10152](info): set up 2 sockets
Aug 06 02:45:44 dist-test-slave-gmjp krb5kdc[10152](info): commencing operation
krb5kdc: starting...
W20260806 02:45:45.698832 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.219s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:45:45 dist-test-slave-gmjp krb5kdc[10152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:45:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
I20260806 02:45:45.839291 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:45877
--webserver_interface=127.1.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:46379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:45877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:45.957115 10170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:45.957466 10170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:45.957551 10170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:45.961779 10170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:45.961905 10170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:45.961943 10170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:45.961984 10170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:45.962016 10170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:45.967623 10170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:46379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:45877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:45877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:45.968889 10170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:45.970009 10170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:45.979032 10178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:45.979022 10179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:45.979197 10170 server_base.cc:1034] running on GCE node
W20260806 02:45:45.979548 10181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:45.980335 10170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:45.981503 10170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:45.982724 10170 hybrid_clock.cc:648] HybridClock initialized: now 1785984345982695 us; error 38 us; skew 500 ppm
Aug 06 02:45:45 dist-test-slave-gmjp krb5kdc[10152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984345, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:45.989770 10170 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:45:45.990908 10170 webserver.cc:460] Webserver started at http://127.1.183.190:44373/ using document root <none> and password file <none>
I20260806 02:45:45.991518 10170 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:45.991586 10170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:45.991776 10170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:45.994354 10170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "e1f25c5bf0814b19bd4038079dda1be1"
format_stamp: "Formatted at 2026-08-06 02:45:45 on dist-test-slave-gmjp"
server_key: "77937afa82fd458b51e57a8d14804777"
server_key_iv: "072f932fa888e18d2634651b3c8c75c9"
server_key_version: "encryptionkey@0"
I20260806 02:45:45.994998 10170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "e1f25c5bf0814b19bd4038079dda1be1"
format_stamp: "Formatted at 2026-08-06 02:45:45 on dist-test-slave-gmjp"
server_key: "77937afa82fd458b51e57a8d14804777"
server_key_iv: "072f932fa888e18d2634651b3c8c75c9"
server_key_version: "encryptionkey@0"
I20260806 02:45:45.999243 10170 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260806 02:45:46.002064 10192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:46.003552 10170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260806 02:45:46.003687 10170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "e1f25c5bf0814b19bd4038079dda1be1"
format_stamp: "Formatted at 2026-08-06 02:45:45 on dist-test-slave-gmjp"
server_key: "77937afa82fd458b51e57a8d14804777"
server_key_iv: "072f932fa888e18d2634651b3c8c75c9"
server_key_version: "encryptionkey@0"
I20260806 02:45:46.003810 10170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:46.057186 10170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:46.061273 10170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:46.061477 10170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:46.070133 10170 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:45877
I20260806 02:45:46.070169 10266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:45877 every 8 connection(s)
I20260806 02:45:46.071269 10170 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:46.074671 10267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:46.078142 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 10170
I20260806 02:45:46.078284 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:45:46.078632 1758 external_mini_cluster.cc:1428] Setting key 5db950d0a8d76fa17bcf50a73eaa6d5d
I20260806 02:45:46.081238 10267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1f25c5bf0814b19bd4038079dda1be1: Bootstrap starting.
I20260806 02:45:46.083770 10267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1f25c5bf0814b19bd4038079dda1be1: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:46.084643 10267 log.cc:824] T 00000000000000000000000000000000 P e1f25c5bf0814b19bd4038079dda1be1: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:46.086700 10267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1f25c5bf0814b19bd4038079dda1be1: No bootstrap required, opened a new log
Aug 06 02:45:46 dist-test-slave-gmjp krb5kdc[10152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:46.090423 10267 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1f25c5bf0814b19bd4038079dda1be1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f25c5bf0814b19bd4038079dda1be1" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 45877 } }
I20260806 02:45:46.090688 10267 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1f25c5bf0814b19bd4038079dda1be1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:46.090777 10267 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1f25c5bf0814b19bd4038079dda1be1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1f25c5bf0814b19bd4038079dda1be1, State: Initialized, Role: FOLLOWER
I20260806 02:45:46.091288 10267 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1f25c5bf0814b19bd4038079dda1be1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f25c5bf0814b19bd4038079dda1be1" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 45877 } }
I20260806 02:45:46.091447 10267 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1f25c5bf0814b19bd4038079dda1be1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:46.091537 10267 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1f25c5bf0814b19bd4038079dda1be1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:46.091641 10267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1f25c5bf0814b19bd4038079dda1be1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:46.092662 10267 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1f25c5bf0814b19bd4038079dda1be1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f25c5bf0814b19bd4038079dda1be1" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 45877 } }
I20260806 02:45:46.093025 10267 leader_election.cc:302] T 00000000000000000000000000000000 P e1f25c5bf0814b19bd4038079dda1be1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1f25c5bf0814b19bd4038079dda1be1; no voters:
I20260806 02:45:46.093400 10267 leader_election.cc:288] T 00000000000000000000000000000000 P e1f25c5bf0814b19bd4038079dda1be1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:46.093638 10273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1f25c5bf0814b19bd4038079dda1be1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:46.093938 10273 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1f25c5bf0814b19bd4038079dda1be1 [term 1 LEADER]: Becoming Leader. State: Replica: e1f25c5bf0814b19bd4038079dda1be1, State: Running, Role: LEADER
I20260806 02:45:46.094357 10273 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1f25c5bf0814b19bd4038079dda1be1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f25c5bf0814b19bd4038079dda1be1" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 45877 } }
I20260806 02:45:46.094713 10271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:46.080285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41462 (local address 127.1.183.190:45877)
0806 02:45:46.080813 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:46.080829 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:46.080871 (+ 42us) server_negotiation.cc:407] Connection header received
0806 02:45:46.081623 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:46.081642 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:46.081991 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:46.082345 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:46.083424 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.084614 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.085769 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.086253 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.086437 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:46.089226 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:46.089232 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:46.089247 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:46.089276 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:46.091255 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:46.091763 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:46.091777 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:46.091782 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:46.091901 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:46.092267 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:46.092271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:46.092274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:46.092704 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:46.092942 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:46.093234 (+ 292us) server_negotiation.cc:299] Negotiation successful
0806 02:45:46.093607 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":148,"threads_started":1}
I20260806 02:45:46.094868 10267 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1f25c5bf0814b19bd4038079dda1be1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:46.096093 10276 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1f25c5bf0814b19bd4038079dda1be1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1f25c5bf0814b19bd4038079dda1be1. Latest consensus state: current_term: 1 leader_uuid: "e1f25c5bf0814b19bd4038079dda1be1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f25c5bf0814b19bd4038079dda1be1" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 45877 } } }
I20260806 02:45:46.096306 10276 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1f25c5bf0814b19bd4038079dda1be1 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:46.097380 10285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:46.097424 10274 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1f25c5bf0814b19bd4038079dda1be1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1f25c5bf0814b19bd4038079dda1be1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f25c5bf0814b19bd4038079dda1be1" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 45877 } } }
I20260806 02:45:46.099133 10274 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1f25c5bf0814b19bd4038079dda1be1 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:46.101367 10285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:46.107692 10285 catalog_manager.cc:1347] Generated new cluster ID: 1297a484b1d24bb5bea8fdf477b96c84
I20260806 02:45:46.107790 10285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:46.120551 10285 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:46.121596 10285 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:46.127882 10285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1f25c5bf0814b19bd4038079dda1be1: Generated new TSK 0
I20260806 02:45:46.128697 10285 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
I20260806 02:45:46.165458 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.129:0
--local_ip_for_outbound_sockets=127.1.183.129
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:45877
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:46379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:46.301316 10309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:46.301676 10309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:46.301759 10309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:46.306185 10309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:46.306314 10309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:46.306414 10309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.129
I20260806 02:45:46.312372 10309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:46379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:45877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.129
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:46.313681 10309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:46.314776 10309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:46.324759 10314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:46.324772 10317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:46.324765 10315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:46.325227 10309 server_base.cc:1034] running on GCE node
I20260806 02:45:46.325675 10309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:46.326375 10309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:46.327586 10309 hybrid_clock.cc:648] HybridClock initialized: now 1785984346327553 us; error 29 us; skew 500 ppm
Aug 06 02:45:46 dist-test-slave-gmjp krb5kdc[10152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984346, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:46.330691 10309 init.cc:377] Logged in from keytab as kudu/127.1.183.129@KRBTEST.COM (short username kudu)
I20260806 02:45:46.331578 10309 webserver.cc:460] Webserver started at http://127.1.183.129:43103/ using document root <none> and password file <none>
I20260806 02:45:46.332213 10309 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:46.332307 10309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:46.332610 10309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:46.335690 10309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-0/data/instance:
uuid: "7c43b0a229c14b38b6d5422a46822c27"
format_stamp: "Formatted at 2026-08-06 02:45:46 on dist-test-slave-gmjp"
server_key: "974cecc24b6666142b14da87f5e2a129"
server_key_iv: "ff6c41b2cbbc1df5fa6194cdedeaf9c9"
server_key_version: "encryptionkey@0"
I20260806 02:45:46.336539 10309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance:
uuid: "7c43b0a229c14b38b6d5422a46822c27"
format_stamp: "Formatted at 2026-08-06 02:45:46 on dist-test-slave-gmjp"
server_key: "974cecc24b6666142b14da87f5e2a129"
server_key_iv: "ff6c41b2cbbc1df5fa6194cdedeaf9c9"
server_key_version: "encryptionkey@0"
I20260806 02:45:46.343317 10309 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 02:45:46.346508 10330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:46.347790 10309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:45:46.347941 10309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-0/wal
uuid: "7c43b0a229c14b38b6d5422a46822c27"
format_stamp: "Formatted at 2026-08-06 02:45:46 on dist-test-slave-gmjp"
server_key: "974cecc24b6666142b14da87f5e2a129"
server_key_iv: "ff6c41b2cbbc1df5fa6194cdedeaf9c9"
server_key_version: "encryptionkey@0"
I20260806 02:45:46.348081 10309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:46.362572 10309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:46.366276 10309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:46.366459 10309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:46.367184 10309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:46.368109 10309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:46.368173 10309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:46.368273 10309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:46.368311 10309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:46.377756 10309 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.129:43647
I20260806 02:45:46.377787 10501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.129:43647 every 8 connection(s)
I20260806 02:45:46.378755 10309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
I20260806 02:45:46.385072 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 10309
I20260806 02:45:46.385229 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance
I20260806 02:45:46.385524 1758 external_mini_cluster.cc:1428] Setting key bd66c6e8614c4c3e013ef0addfc88b03
Aug 06 02:45:46 dist-test-slave-gmjp krb5kdc[10152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984346, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:46.394124 10271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:46.380692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:56071 (local address 127.1.183.190:45877)
0806 02:45:46.380836 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:46.380840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:46.381800 (+ 960us) server_negotiation.cc:407] Connection header received
0806 02:45:46.383517 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:46.383520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:46.383572 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:46.383651 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:46.385015 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.385705 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.387125 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.387363 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.387478 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:46.390122 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:46.390126 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:46.390129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:46.390154 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:46.391818 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:46.392459 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:46.392463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:46.392465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:46.392554 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:46.392935 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:46.392939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:46.392940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:46.393137 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:46.393244 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:46.393808 (+ 564us) server_negotiation.cc:299] Negotiation successful
0806 02:45:46.393949 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 02:45:46.395115 10504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:46.381034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:45877 (local address 127.1.183.129:56071)
0806 02:45:46.381579 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:46.381650 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:46.383293 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:46.383832 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:46.383839 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:46.384297 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:46.384841 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:46.384858 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.385848 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.385856 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:46.386991 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:46.387001 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.387525 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.387530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:46.387705 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:46.388390 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:46.388409 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:46.389930 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:46.391943 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:46.391950 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:46.391968 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:46.392349 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:46.392655 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:46.392659 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:46.392661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:46.392815 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:46.393262 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:46.393269 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:46.393547 (+ 278us) client_negotiation.cc:769] Sending connection context
0806 02:45:46.393747 (+ 200us) client_negotiation.cc:240] Negotiation successful
0806 02:45:46.394119 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.1.183.130@KRBTEST.COM; defaulting to no policy
I20260806 02:45:46.396541 10502 heartbeater.cc:347] Connected to a master server at 127.1.183.190:45877
I20260806 02:45:46.396819 10502 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:46.397346 10502 heartbeater.cc:511] Master 127.1.183.190:45877 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.130@KRBTEST.COM" created.
I20260806 02:45:46.399178 10219 ts_manager.cc:194] Registered new tserver with Master: 7c43b0a229c14b38b6d5422a46822c27 (127.1.183.129:43647)
I20260806 02:45:46.400487 10219 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.129@KRBTEST.COM'} at 127.1.183.129:56071
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
I20260806 02:45:46.416918 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.130:0
--local_ip_for_outbound_sockets=127.1.183.130
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:45877
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:46379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:46.536798 10509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:46.537072 10509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:46.537145 10509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:46.541340 10509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:46.541411 10509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:46.541513 10509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.130
I20260806 02:45:46.546978 10509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:46379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:45877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.130
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:46.548138 10509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:46.549249 10509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:46.559876 10522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:46.559876 10517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:46.559952 10509 server_base.cc:1034] running on GCE node
W20260806 02:45:46.559876 10519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:46.560655 10509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:46.561357 10509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:46.562577 10509 hybrid_clock.cc:648] HybridClock initialized: now 1785984346562557 us; error 31 us; skew 500 ppm
Aug 06 02:45:46 dist-test-slave-gmjp krb5kdc[10152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984346, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:46.565856 10509 init.cc:377] Logged in from keytab as kudu/127.1.183.130@KRBTEST.COM (short username kudu)
I20260806 02:45:46.566811 10509 webserver.cc:460] Webserver started at http://127.1.183.130:46563/ using document root <none> and password file <none>
I20260806 02:45:46.567395 10509 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:46.567457 10509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:46.567633 10509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:46.569849 10509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-1/data/instance:
uuid: "1ab4391e34fd4bbf9697c137e04c349d"
format_stamp: "Formatted at 2026-08-06 02:45:46 on dist-test-slave-gmjp"
server_key: "d28f63e30529c946294096c526d92e11"
server_key_iv: "bb61cc2d14aa7c24acb89122fc0800c5"
server_key_version: "encryptionkey@0"
I20260806 02:45:46.570389 10509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance:
uuid: "1ab4391e34fd4bbf9697c137e04c349d"
format_stamp: "Formatted at 2026-08-06 02:45:46 on dist-test-slave-gmjp"
server_key: "d28f63e30529c946294096c526d92e11"
server_key_iv: "bb61cc2d14aa7c24acb89122fc0800c5"
server_key_version: "encryptionkey@0"
I20260806 02:45:46.574339 10509 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260806 02:45:46.576933 10530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:46.578277 10509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:45:46.578428 10509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-1/wal
uuid: "1ab4391e34fd4bbf9697c137e04c349d"
format_stamp: "Formatted at 2026-08-06 02:45:46 on dist-test-slave-gmjp"
server_key: "d28f63e30529c946294096c526d92e11"
server_key_iv: "bb61cc2d14aa7c24acb89122fc0800c5"
server_key_version: "encryptionkey@0"
I20260806 02:45:46.578570 10509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:46.592561 10509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:46.596458 10509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:46.596694 10509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:46.597417 10509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:46.598387 10509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:46.598481 10509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:46.598565 10509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:46.598623 10509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:46.609256 10509 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.130:37703
I20260806 02:45:46.609269 10689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.130:37703 every 8 connection(s)
I20260806 02:45:46.610340 10509 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
I20260806 02:45:46.614897 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 10509
I20260806 02:45:46.615031 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance
I20260806 02:45:46.615381 1758 external_mini_cluster.cc:1428] Setting key f8a549c92f03e36c036abcef0cf3043b
Aug 06 02:45:46 dist-test-slave-gmjp krb5kdc[10152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984346, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:46.626394 10271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:46.612462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:49769 (local address 127.1.183.190:45877)
0806 02:45:46.612634 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:46.612639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:46.613602 (+ 963us) server_negotiation.cc:407] Connection header received
0806 02:45:46.615149 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:46.615153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:46.615213 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:46.615275 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:46.616833 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.617373 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.618565 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.618931 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.619047 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:46.622082 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:46.622086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:46.622088 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:46.622113 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:46.623891 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:46.624536 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:46.624542 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:46.624547 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:46.624644 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:46.625036 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:46.625040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:46.625042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:46.625322 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:46.625425 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:46.626033 (+ 608us) server_negotiation.cc:299] Negotiation successful
0806 02:45:46.626170 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
WARNING: no policy specified for kudu/127.1.183.131@KRBTEST.COM; defaulting to no policy
I20260806 02:45:46.627355 10692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:46.612964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:45877 (local address 127.1.183.130:49769)
0806 02:45:46.613441 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:46.613485 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:46.614878 (+ 1393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:46.615709 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:46.615718 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:46.616111 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:46.616680 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:46.616694 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.617527 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.617529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:46.618426 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:46.618435 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.619094 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.619098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:46.619270 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:46.619940 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:46.619957 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:46.621889 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:46.624046 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:46.624051 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:46.624065 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:46.624427 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:46.624746 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:46.624748 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:46.624749 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:46.624920 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:46.625471 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:46.625478 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:46.625738 (+ 260us) client_negotiation.cc:769] Sending connection context
0806 02:45:46.625908 (+ 170us) client_negotiation.cc:240] Negotiation successful
0806 02:45:46.626180 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":120,"threads_started":1}
I20260806 02:45:46.628743 10690 heartbeater.cc:347] Connected to a master server at 127.1.183.190:45877
I20260806 02:45:46.628985 10690 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:46.629586 10690 heartbeater.cc:511] Master 127.1.183.190:45877 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.131@KRBTEST.COM" created.
I20260806 02:45:46.630839 10219 ts_manager.cc:194] Registered new tserver with Master: 1ab4391e34fd4bbf9697c137e04c349d (127.1.183.130:37703)
I20260806 02:45:46.631465 10219 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.130@KRBTEST.COM'} at 127.1.183.130:49769
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
I20260806 02:45:46.647183 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.131:0
--local_ip_for_outbound_sockets=127.1.183.131
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:45877
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:46379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:46.766469 10697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:46.766741 10697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:46.766785 10697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:46.770840 10697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:46.770901 10697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:46.770961 10697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.131
I20260806 02:45:46.776881 10697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:46379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:45877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.131
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:46.777982 10697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:46.779105 10697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:46.789314 10705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:46.789362 10704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:46.789351 10711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:46.789880 10697 server_base.cc:1034] running on GCE node
I20260806 02:45:46.790313 10697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:46.791072 10697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:46.792230 10697 hybrid_clock.cc:648] HybridClock initialized: now 1785984346792204 us; error 32 us; skew 500 ppm
Aug 06 02:45:46 dist-test-slave-gmjp krb5kdc[10152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984346, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:46.795197 10697 init.cc:377] Logged in from keytab as kudu/127.1.183.131@KRBTEST.COM (short username kudu)
I20260806 02:45:46.796417 10697 webserver.cc:460] Webserver started at http://127.1.183.131:37491/ using document root <none> and password file <none>
I20260806 02:45:46.797058 10697 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:46.797155 10697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:46.797401 10697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:46.799635 10697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-2/data/instance:
uuid: "0993018c88714530b6cd16756521bbf7"
format_stamp: "Formatted at 2026-08-06 02:45:46 on dist-test-slave-gmjp"
server_key: "0411609d488f9971ebd7c08a29dad122"
server_key_iv: "c7c61f8714165821e78baf867ca17597"
server_key_version: "encryptionkey@0"
I20260806 02:45:46.800311 10697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance:
uuid: "0993018c88714530b6cd16756521bbf7"
format_stamp: "Formatted at 2026-08-06 02:45:46 on dist-test-slave-gmjp"
server_key: "0411609d488f9971ebd7c08a29dad122"
server_key_iv: "c7c61f8714165821e78baf867ca17597"
server_key_version: "encryptionkey@0"
I20260806 02:45:46.804453 10697 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:45:46.807039 10721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:46.808501 10697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:46.808643 10697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-2/wal
uuid: "0993018c88714530b6cd16756521bbf7"
format_stamp: "Formatted at 2026-08-06 02:45:46 on dist-test-slave-gmjp"
server_key: "0411609d488f9971ebd7c08a29dad122"
server_key_iv: "c7c61f8714165821e78baf867ca17597"
server_key_version: "encryptionkey@0"
I20260806 02:45:46.808780 10697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:46.837138 10697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:46.840585 10697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:46.840816 10697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:46.841477 10697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:46.842399 10697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:46.842475 10697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:46.842549 10697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:46.842597 10697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:46.851851 10697 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.131:42677
I20260806 02:45:46.851871 10885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.131:42677 every 8 connection(s)
I20260806 02:45:46.852923 10697 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
I20260806 02:45:46.855435 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 10697
I20260806 02:45:46.855573 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance
I20260806 02:45:46.855921 1758 external_mini_cluster.cc:1428] Setting key 2e3b4ab762a5b35bc1fdeaa003f0fb08
Aug 06 02:45:46 dist-test-slave-gmjp krb5kdc[10152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984346, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:45:46.868173 10271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:46.854838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:51435 (local address 127.1.183.190:45877)
0806 02:45:46.855043 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:46.855048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:46.856084 (+ 1036us) server_negotiation.cc:407] Connection header received
0806 02:45:46.857840 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:46.857843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:46.857900 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:46.858011 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:46.859400 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.859948 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.861164 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.861490 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.861608 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:46.864338 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:46.864347 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:46.864349 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:46.864376 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:46.865920 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:46.866496 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:46.866500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:46.866501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:46.866589 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:46.866970 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:46.866973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:46.866975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:46.867173 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:46.867285 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:46.867838 (+ 553us) server_negotiation.cc:299] Negotiation successful
0806 02:45:46.867974 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260806 02:45:46.869071 10889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:46.855308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:45877 (local address 127.1.183.131:51435)
0806 02:45:46.855908 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:46.855955 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:46.857618 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:46.858341 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:46.858349 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:46.858736 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:46.859239 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:46.859253 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.860085 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.860089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:46.861030 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:46.861040 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.861656 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.861660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:46.861842 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:46.862521 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:46.862539 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:46.864066 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:46.866064 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:46.866071 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:46.866090 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:46.866377 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:46.866689 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:46.866692 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:46.866694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:46.866860 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:46.867287 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:46.867297 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:46.867579 (+ 282us) client_negotiation.cc:769] Sending connection context
0806 02:45:46.867779 (+ 200us) client_negotiation.cc:240] Negotiation successful
0806 02:45:46.868146 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":156,"threads_started":1}
I20260806 02:45:46.870236 10887 heartbeater.cc:347] Connected to a master server at 127.1.183.190:45877
I20260806 02:45:46.870527 10887 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:46.871013 10887 heartbeater.cc:511] Master 127.1.183.190:45877 requested a full tablet report, sending...
I20260806 02:45:46.872008 10219 ts_manager.cc:194] Registered new tserver with Master: 0993018c88714530b6cd16756521bbf7 (127.1.183.131:42677)
I20260806 02:45:46.872640 10219 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.131@KRBTEST.COM'} at 127.1.183.131:51435
I20260806 02:45:46.882522 1758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/test-admin.keytab.
Aug 06 02:45:46 dist-test-slave-gmjp krb5kdc[10152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:46.901624 1758 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260806 02:45:46.904805 10905 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:46.903122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37358 (local address 127.1.183.129:43647)
0806 02:45:46.903452 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:46.903460 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:46.903489 (+ 29us) server_negotiation.cc:407] Connection header received
0806 02:45:46.903616 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:46.903621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:46.903818 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:46.903984 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:46.904593 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37358: BlockingRecv error: recv got EOF from 127.0.0.1:37358 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":87,"threads_started":1}
W20260806 02:45:46.904927 1758 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/krb5cc' of type FILE
Aug 06 02:45:46 dist-test-slave-gmjp krb5kdc[10152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:46.906041 1758 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 02:45:46 dist-test-slave-gmjp krb5kdc[10152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:46.907027 1758 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 02:45:46 dist-test-slave-gmjp krb5kdc[10152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.129@KRBTEST.COM
I20260806 02:45:46.917915 10905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:46.908084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37374 (local address 127.1.183.129:43647)
0806 02:45:46.908256 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:46.908261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:46.908387 (+ 126us) server_negotiation.cc:407] Connection header received
0806 02:45:46.908553 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:46.908556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:46.908609 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:46.908694 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:46.909273 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.910042 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.910946 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.911354 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.911484 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:46.914129 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:46.914141 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:46.914165 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:46.914193 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:46.915944 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:46.916502 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:46.916507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:46.916509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:46.916591 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:46.916891 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:46.916895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:46.916897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:46.917181 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:46.917439 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:46.917605 (+ 166us) server_negotiation.cc:299] Negotiation successful
0806 02:45:46.917762 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260806 02:45:46.922058 10905 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:46.920949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37388 (local address 127.1.183.129:43647)
0806 02:45:46.921092 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:46.921096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:46.921206 (+ 110us) server_negotiation.cc:407] Connection header received
0806 02:45:46.921359 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:46.921369 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:46.921418 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:46.921503 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:46.921944 (+ 441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37388: BlockingRecv error: recv got EOF from 127.0.0.1:37388 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260806 02:45:46.922187 1758 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/krb5cc' of type FILE
Aug 06 02:45:46 dist-test-slave-gmjp krb5kdc[10152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:46.923274 1758 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 02:45:46 dist-test-slave-gmjp krb5kdc[10152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:46.924235 1758 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 02:45:46 dist-test-slave-gmjp krb5kdc[10152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.129@KRBTEST.COM
I20260806 02:45:46.933349 10905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:46.925124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37390 (local address 127.1.183.129:43647)
0806 02:45:46.925254 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:46.925258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:46.925361 (+ 103us) server_negotiation.cc:407] Connection header received
0806 02:45:46.925511 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:46.925514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:46.925558 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:46.925643 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:46.926156 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.926673 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.927492 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.927733 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.927838 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:46.930141 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:46.930145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:46.930148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:46.930174 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:46.931632 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:46.932073 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:46.932077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:46.932079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:46.932231 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:46.932509 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:46.932513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:46.932515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:46.932687 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:46.932850 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:46.933072 (+ 222us) server_negotiation.cc:299] Negotiation successful
0806 02:45:46.933177 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260806 02:45:46.935894 10905 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:46.935085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37398 (local address 127.1.183.129:43647)
0806 02:45:46.935226 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:46.935229 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:46.935273 (+ 44us) server_negotiation.cc:407] Connection header received
0806 02:45:46.935364 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:46.935366 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:46.935401 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:46.935470 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:46.935807 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37398: BlockingRecv error: recv got EOF from 127.0.0.1:37398 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260806 02:45:46.935992 1758 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984313887056-1758-0/krb5kdc/krb5cc' of type FILE
Aug 06 02:45:46 dist-test-slave-gmjp krb5kdc[10152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:46.937013 1758 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 02:45:46 dist-test-slave-gmjp krb5kdc[10152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:46.937937 1758 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 02:45:46 dist-test-slave-gmjp krb5kdc[10152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.129@KRBTEST.COM
I20260806 02:45:46.946401 10905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:46.938756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37408 (local address 127.1.183.129:43647)
0806 02:45:46.938891 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:46.938895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:46.938964 (+ 69us) server_negotiation.cc:407] Connection header received
0806 02:45:46.939093 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:46.939103 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:46.939138 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:46.939220 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:46.939708 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.940269 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.941063 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:46.941276 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:46.941386 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:46.943618 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:46.943621 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:46.943624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:46.943644 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:46.945127 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:46.945482 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:46.945484 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:46.945486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:46.945554 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:46.945779 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:46.945781 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:46.945782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:46.945952 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:46.946059 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:46.946209 (+ 150us) server_negotiation.cc:299] Negotiation successful
0806 02:45:46.946290 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:45:46.947573 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 10309
I20260806 02:45:46.962381 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 10509
I20260806 02:45:46.974588 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 10697
I20260806 02:45:46.991456 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 10170
2026-08-06T02:45:47Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2541 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:45:47 dist-test-slave-gmjp krb5kdc[10919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:45:47 dist-test-slave-gmjp krb5kdc[10919](info): set up 2 sockets
Aug 06 02:45:47 dist-test-slave-gmjp krb5kdc[10919](info): commencing operation
krb5kdc: starting...
W20260806 02:45:48.226830 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.205s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:45:48 dist-test-slave-gmjp krb5kdc[10919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:45:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:48Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.190.keytab.
Entry for principal oryx/127.1.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.190.keytab.
I20260806 02:45:48.359252 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:39091
--webserver_interface=127.1.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.190.keytab
--principal=oryx/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:39639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:39091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:48.479833 10940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:48.480216 10940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:48.480302 10940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:48.484488 10940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:48.484580 10940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:48.484618 10940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:48.484659 10940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:48.484696 10940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:48.490197 10940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:39639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:39091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.190.keytab
--principal=oryx/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:39091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:48.491410 10940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:48.492579 10940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:48.501897 10948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:48.501847 10947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:48.501948 10950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:48.502313 10940 server_base.cc:1034] running on GCE node
I20260806 02:45:48.502962 10940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:48.504144 10940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:48.505376 10940 hybrid_clock.cc:648] HybridClock initialized: now 1785984348505361 us; error 27 us; skew 500 ppm
Aug 06 02:45:48 dist-test-slave-gmjp krb5kdc[10919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984348, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:48.508371 10940 init.cc:377] Logged in from keytab as oryx/127.1.183.190@KRBTEST.COM (short username oryx)
I20260806 02:45:48.509387 10940 webserver.cc:460] Webserver started at http://127.1.183.190:32847/ using document root <none> and password file <none>
I20260806 02:45:48.509994 10940 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:48.510092 10940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:48.510335 10940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:48.512663 10940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "5975de0199eb4bc1a2ac5d5607e1928c"
format_stamp: "Formatted at 2026-08-06 02:45:48 on dist-test-slave-gmjp"
server_key: "2f9c80d763fa5e76d31ae0efcbeaa881"
server_key_iv: "b03d56aa7a3f1a70b9042ac46c795a10"
server_key_version: "encryptionkey@0"
I20260806 02:45:48.513260 10940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "5975de0199eb4bc1a2ac5d5607e1928c"
format_stamp: "Formatted at 2026-08-06 02:45:48 on dist-test-slave-gmjp"
server_key: "2f9c80d763fa5e76d31ae0efcbeaa881"
server_key_iv: "b03d56aa7a3f1a70b9042ac46c795a10"
server_key_version: "encryptionkey@0"
I20260806 02:45:48.517364 10940 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:45:48.519969 10962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:48.521337 10940 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:45:48.521503 10940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "5975de0199eb4bc1a2ac5d5607e1928c"
format_stamp: "Formatted at 2026-08-06 02:45:48 on dist-test-slave-gmjp"
server_key: "2f9c80d763fa5e76d31ae0efcbeaa881"
server_key_iv: "b03d56aa7a3f1a70b9042ac46c795a10"
server_key_version: "encryptionkey@0"
I20260806 02:45:48.521631 10940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:48.541275 10940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:48.545035 10940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:48.545274 10940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:48.552997 10940 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:39091
I20260806 02:45:48.553308 11027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:39091 every 8 connection(s)
I20260806 02:45:48.554080 10940 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:48.557024 11029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:48.557432 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 10940
I20260806 02:45:48.557588 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:45:48.557900 1758 external_mini_cluster.cc:1428] Setting key 05b6aafd49d0745cf930cac5e1c082ab
I20260806 02:45:48.564106 11029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c: Bootstrap starting.
I20260806 02:45:48.566570 11029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c: Neither blocks nor log segments found. Creating new log.
Aug 06 02:45:48 dist-test-slave-gmjp krb5kdc[10919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM
I20260806 02:45:48.567548 11029 log.cc:824] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:48.569835 11029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c: No bootstrap required, opened a new log
I20260806 02:45:48.572862 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:48.559523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53528 (local address 127.1.183.190:39091)
0806 02:45:48.559989 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:48.560004 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:48.560042 (+ 38us) server_negotiation.cc:407] Connection header received
0806 02:45:48.560834 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:48.560853 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:48.561194 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:48.561549 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:48.562411 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:48.563405 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:48.564330 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:48.564817 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:48.564990 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:48.567876 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:48.567882 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:48.567904 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:48.567935 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:48.569842 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:48.570399 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:48.570404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:48.570409 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:48.570530 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:48.570848 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:48.570850 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:48.570851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:48.571255 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:48.571417 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:48.571703 (+ 286us) server_negotiation.cc:299] Negotiation successful
0806 02:45:48.572018 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":127,"threads_started":1}
I20260806 02:45:48.574186 11029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } }
I20260806 02:45:48.574469 11029 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:48.574533 11029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5975de0199eb4bc1a2ac5d5607e1928c, State: Initialized, Role: FOLLOWER
I20260806 02:45:48.575011 11029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } }
I20260806 02:45:48.575199 11029 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:48.575302 11029 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:48.575412 11029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:48.576529 11029 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } }
I20260806 02:45:48.576913 11029 leader_election.cc:302] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5975de0199eb4bc1a2ac5d5607e1928c; no voters:
I20260806 02:45:48.577283 11029 leader_election.cc:288] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:48.577412 11034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:48.577654 11034 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [term 1 LEADER]: Becoming Leader. State: Replica: 5975de0199eb4bc1a2ac5d5607e1928c, State: Running, Role: LEADER
I20260806 02:45:48.578060 11034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } }
I20260806 02:45:48.578718 11029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:48.579803 11036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } }
I20260806 02:45:48.579950 11036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:48.580353 11048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:48.580652 11037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5975de0199eb4bc1a2ac5d5607e1928c. Latest consensus state: current_term: 1 leader_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } }
I20260806 02:45:48.580826 11037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:48.583278 11048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:48.589207 11048 catalog_manager.cc:1347] Generated new cluster ID: 07ac2c30b95046aa9ad9b2c2891f35ca
I20260806 02:45:48.589290 11048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:48.598699 11048 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:48.600041 11048 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:48.611670 11048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c: Generated new TSK 0
I20260806 02:45:48.612671 11048 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.129.keytab.
Entry for principal oryx/127.1.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.129.keytab.
I20260806 02:45:48.644012 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.129:0
--local_ip_for_outbound_sockets=127.1.183.129
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:39091
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.129.keytab
--principal=oryx/127.1.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:39639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:48.762837 11067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:48.763204 11067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:48.763288 11067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:48.767407 11067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:48.767525 11067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:48.767622 11067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.129
I20260806 02:45:48.773557 11067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:39639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.129.keytab
--principal=oryx/127.1.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:39091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.129
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:48.774775 11067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:48.775897 11067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:48.786417 11076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:48.786417 11082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:48.786496 11067 server_base.cc:1034] running on GCE node
W20260806 02:45:48.786432 11074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:48.787112 11067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:48.787864 11067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:48.789072 11067 hybrid_clock.cc:648] HybridClock initialized: now 1785984348789044 us; error 42 us; skew 500 ppm
Aug 06 02:45:48 dist-test-slave-gmjp krb5kdc[10919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984348, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:48.792007 11067 init.cc:377] Logged in from keytab as oryx/127.1.183.129@KRBTEST.COM (short username oryx)
I20260806 02:45:48.793309 11067 webserver.cc:460] Webserver started at http://127.1.183.129:45685/ using document root <none> and password file <none>
I20260806 02:45:48.793951 11067 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:48.794018 11067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:48.794252 11067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:48.796514 11067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-0/data/instance:
uuid: "fff12eae637c41ebabb8689441a6dbe5"
format_stamp: "Formatted at 2026-08-06 02:45:48 on dist-test-slave-gmjp"
server_key: "7c6bca1819a9a888efb1fa1f4ce94286"
server_key_iv: "fd43a0fa4ce5cb0f32286dbdcfe33003"
server_key_version: "encryptionkey@0"
I20260806 02:45:48.797148 11067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance:
uuid: "fff12eae637c41ebabb8689441a6dbe5"
format_stamp: "Formatted at 2026-08-06 02:45:48 on dist-test-slave-gmjp"
server_key: "7c6bca1819a9a888efb1fa1f4ce94286"
server_key_iv: "fd43a0fa4ce5cb0f32286dbdcfe33003"
server_key_version: "encryptionkey@0"
I20260806 02:45:48.801218 11067 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:45:48.803781 11093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:48.805166 11067 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 02:45:48.805320 11067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-0/wal
uuid: "fff12eae637c41ebabb8689441a6dbe5"
format_stamp: "Formatted at 2026-08-06 02:45:48 on dist-test-slave-gmjp"
server_key: "7c6bca1819a9a888efb1fa1f4ce94286"
server_key_iv: "fd43a0fa4ce5cb0f32286dbdcfe33003"
server_key_version: "encryptionkey@0"
I20260806 02:45:48.805464 11067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:48.816337 11067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:48.819450 11067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:48.819621 11067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:48.820309 11067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:48.821231 11067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:48.821282 11067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:48.821326 11067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:48.821349 11067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:48.831171 11067 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.129:34391
I20260806 02:45:48.831189 11261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.129:34391 every 8 connection(s)
I20260806 02:45:48.832253 11067 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
Aug 06 02:45:48 dist-test-slave-gmjp krb5kdc[10919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984348, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM
I20260806 02:45:48.842021 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 11067
I20260806 02:45:48.842164 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance
I20260806 02:45:48.842468 1758 external_mini_cluster.cc:1428] Setting key 5641e032338382a2c59bd03566c368ac
I20260806 02:45:48.846376 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:48.834176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:52449 (local address 127.1.183.190:39091)
0806 02:45:48.834316 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:48.834320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:48.835384 (+ 1064us) server_negotiation.cc:407] Connection header received
0806 02:45:48.836330 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:48.836333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:48.836384 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:48.836460 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:48.837632 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:48.838160 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:48.839265 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:48.839506 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:48.839615 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:48.842341 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:48.842345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:48.842347 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:48.842372 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:48.843945 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:48.844602 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:48.844605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:48.844606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:48.844683 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:48.845095 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:48.845097 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:48.845099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:48.845272 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:48.845350 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:48.846012 (+ 662us) server_negotiation.cc:299] Negotiation successful
0806 02:45:48.846122 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:45:48.847329 11264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:48.834546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.1.183.129:52449)
0806 02:45:48.835232 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:48.835274 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:48.836075 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:48.836624 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:48.836631 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:48.837013 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:48.837499 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:48.837512 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:48.838271 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:48.838273 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:48.839161 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:48.839169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:48.839651 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:48.839654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:48.839793 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:48.840504 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:48.840521 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:48.842136 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:48.844058 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:48.844065 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:48.844082 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:48.844503 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:48.844776 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:48.844781 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:48.844786 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:48.844994 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:48.845373 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:48.845382 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:48.845726 (+ 344us) client_negotiation.cc:769] Sending connection context
0806 02:45:48.845952 (+ 226us) client_negotiation.cc:240] Negotiation successful
0806 02:45:48.846336 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":122,"threads_started":1}
I20260806 02:45:48.848656 11262 heartbeater.cc:347] Connected to a master server at 127.1.183.190:39091
I20260806 02:45:48.848958 11262 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:48.849624 11262 heartbeater.cc:511] Master 127.1.183.190:39091 requested a full tablet report, sending...
I20260806 02:45:48.851533 10983 ts_manager.cc:194] Registered new tserver with Master: fff12eae637c41ebabb8689441a6dbe5 (127.1.183.129:34391)
I20260806 02:45:48.853328 10983 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.183.129@KRBTEST.COM'} at 127.1.183.129:52449
WARNING: no policy specified for oryx/127.1.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.130.keytab.
Entry for principal oryx/127.1.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.130.keytab.
I20260806 02:45:48.874851 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.130:0
--local_ip_for_outbound_sockets=127.1.183.130
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:39091
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.130.keytab
--principal=oryx/127.1.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:39639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:48.990801 11268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:48.991087 11268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:48.991133 11268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:48.995276 11268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:48.995357 11268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:48.995474 11268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.130
I20260806 02:45:49.001039 11268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:39639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.130.keytab
--principal=oryx/127.1.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:39091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.130
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:49.002262 11268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:49.003393 11268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:49.013657 11279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:49.013661 11275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:49.013657 11276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:49.013952 11268 server_base.cc:1034] running on GCE node
I20260806 02:45:49.014477 11268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:49.015204 11268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:49.016381 11268 hybrid_clock.cc:648] HybridClock initialized: now 1785984349016358 us; error 32 us; skew 500 ppm
Aug 06 02:45:49 dist-test-slave-gmjp krb5kdc[10919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984349, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:49.019306 11268 init.cc:377] Logged in from keytab as oryx/127.1.183.130@KRBTEST.COM (short username oryx)
I20260806 02:45:49.020367 11268 webserver.cc:460] Webserver started at http://127.1.183.130:46091/ using document root <none> and password file <none>
I20260806 02:45:49.020925 11268 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:49.020987 11268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:49.021162 11268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:49.023202 11268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-1/data/instance:
uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2"
format_stamp: "Formatted at 2026-08-06 02:45:49 on dist-test-slave-gmjp"
server_key: "db8ed42e1ceeaa1a4f890e40c3bd5b62"
server_key_iv: "2a0ba49b2ab6c82de1071f668c227ee6"
server_key_version: "encryptionkey@0"
I20260806 02:45:49.023720 11268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance:
uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2"
format_stamp: "Formatted at 2026-08-06 02:45:49 on dist-test-slave-gmjp"
server_key: "db8ed42e1ceeaa1a4f890e40c3bd5b62"
server_key_iv: "2a0ba49b2ab6c82de1071f668c227ee6"
server_key_version: "encryptionkey@0"
I20260806 02:45:49.027757 11268 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:45:49.030395 11287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:49.031576 11268 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 02:45:49.031718 11268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-1/wal
uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2"
format_stamp: "Formatted at 2026-08-06 02:45:49 on dist-test-slave-gmjp"
server_key: "db8ed42e1ceeaa1a4f890e40c3bd5b62"
server_key_iv: "2a0ba49b2ab6c82de1071f668c227ee6"
server_key_version: "encryptionkey@0"
I20260806 02:45:49.031858 11268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:49.066208 11268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:49.069797 11268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:49.070043 11268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:49.070700 11268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:49.071575 11268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:49.071648 11268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:49.071723 11268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:49.071774 11268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:49.081044 11268 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.130:33151
I20260806 02:45:49.081099 11435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.130:33151 every 8 connection(s)
I20260806 02:45:49.082008 11268 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
I20260806 02:45:49.083287 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 11268
I20260806 02:45:49.083405 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance
I20260806 02:45:49.083734 1758 external_mini_cluster.cc:1428] Setting key f1a4fe0436c4803065a3246ae9977148
Aug 06 02:45:49 dist-test-slave-gmjp krb5kdc[10919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984349, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.1.183.131@KRBTEST.COM; defaulting to no policy
I20260806 02:45:49.097630 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.084397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:40605 (local address 127.1.183.190:39091)
0806 02:45:49.084560 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:49.084564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:49.085559 (+ 995us) server_negotiation.cc:407] Connection header received
0806 02:45:49.086430 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:49.086437 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:49.086502 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:49.086601 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:49.088004 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.088608 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.090387 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.090661 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.090773 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:49.093775 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:49.093779 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:49.093781 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:49.093807 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:49.095428 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:49.095991 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:49.095994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:49.095995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:49.096070 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:49.096462 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:49.096467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:49.096469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:49.096661 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:49.096774 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:49.097323 (+ 549us) server_negotiation.cc:299] Negotiation successful
0806 02:45:49.097457 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.183.131@KRBTEST.COM" created.
I20260806 02:45:49.098721 11440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.084559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.1.183.130:40605)
0806 02:45:49.085377 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:49.085438 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:49.086227 (+ 789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:49.086759 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:49.086765 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:49.087146 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:49.087829 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:49.087851 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.088741 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.088748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:49.090252 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:49.090268 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.090854 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.090861 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:49.091088 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:49.091726 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:49.091753 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:49.093581 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:49.095560 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:49.095567 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:49.095584 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:49.095862 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:49.096188 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:49.096192 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:49.096194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:49.096359 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:49.096774 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:49.096783 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:49.097062 (+ 279us) client_negotiation.cc:769] Sending connection context
0806 02:45:49.097261 (+ 199us) client_negotiation.cc:240] Negotiation successful
0806 02:45:49.097630 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":104,"threads_started":1}
I20260806 02:45:49.100126 11437 heartbeater.cc:347] Connected to a master server at 127.1.183.190:39091
I20260806 02:45:49.100440 11437 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:49.100993 11437 heartbeater.cc:511] Master 127.1.183.190:39091 requested a full tablet report, sending...
I20260806 02:45:49.102135 10983 ts_manager.cc:194] Registered new tserver with Master: 5e8dc07e3a1f4de49b0e6a5bec6878f2 (127.1.183.130:33151)
I20260806 02:45:49.103041 10983 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.183.130@KRBTEST.COM'} at 127.1.183.130:40605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.131.keytab.
Entry for principal oryx/127.1.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.131.keytab.
I20260806 02:45:49.114853 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.131:0
--local_ip_for_outbound_sockets=127.1.183.131
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:39091
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.131.keytab
--principal=oryx/127.1.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:39639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:49.238291 11444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:49.238597 11444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:49.238641 11444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:49.242825 11444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:49.242888 11444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:49.242947 11444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.131
I20260806 02:45:49.248198 11444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:39639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.131.keytab
--principal=oryx/127.1.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:39091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.131
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:49.249264 11444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:49.250252 11444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:49.260211 11453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:49.260260 11456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:49.260218 11454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:49.260275 11444 server_base.cc:1034] running on GCE node
I20260806 02:45:49.260950 11444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:49.261574 11444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:49.262776 11444 hybrid_clock.cc:648] HybridClock initialized: now 1785984349262742 us; error 45 us; skew 500 ppm
Aug 06 02:45:49 dist-test-slave-gmjp krb5kdc[10919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984349, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:49.265631 11444 init.cc:377] Logged in from keytab as oryx/127.1.183.131@KRBTEST.COM (short username oryx)
I20260806 02:45:49.266467 11444 webserver.cc:460] Webserver started at http://127.1.183.131:36435/ using document root <none> and password file <none>
I20260806 02:45:49.267004 11444 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:49.267063 11444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:49.267230 11444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:49.269352 11444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-2/data/instance:
uuid: "e2cbd8619d9048b5b56e4d9ea417288e"
format_stamp: "Formatted at 2026-08-06 02:45:49 on dist-test-slave-gmjp"
server_key: "d1a8b79dca2874c0395e30afffec4812"
server_key_iv: "b3611f31a0682499d1b0562c3457a30f"
server_key_version: "encryptionkey@0"
I20260806 02:45:49.269888 11444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance:
uuid: "e2cbd8619d9048b5b56e4d9ea417288e"
format_stamp: "Formatted at 2026-08-06 02:45:49 on dist-test-slave-gmjp"
server_key: "d1a8b79dca2874c0395e30afffec4812"
server_key_iv: "b3611f31a0682499d1b0562c3457a30f"
server_key_version: "encryptionkey@0"
I20260806 02:45:49.273612 11444 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:45:49.275976 11471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:49.277258 11444 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:45:49.277360 11444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-2/wal
uuid: "e2cbd8619d9048b5b56e4d9ea417288e"
format_stamp: "Formatted at 2026-08-06 02:45:49 on dist-test-slave-gmjp"
server_key: "d1a8b79dca2874c0395e30afffec4812"
server_key_iv: "b3611f31a0682499d1b0562c3457a30f"
server_key_version: "encryptionkey@0"
I20260806 02:45:49.277449 11444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:49.303118 11444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:49.306340 11444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:49.306576 11444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:49.307240 11444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:49.308135 11444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:49.308251 11444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:49.308336 11444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:49.308375 11444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:49.317955 11444 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.131:39727
I20260806 02:45:49.317977 11626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.131:39727 every 8 connection(s)
I20260806 02:45:49.318917 11444 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
I20260806 02:45:49.322731 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 11444
I20260806 02:45:49.322867 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance
I20260806 02:45:49.323172 1758 external_mini_cluster.cc:1428] Setting key fb829db7e0025eea13741a85d5c66238
Aug 06 02:45:49 dist-test-slave-gmjp krb5kdc[10919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984349, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM
I20260806 02:45:49.332628 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.320893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:43145 (local address 127.1.183.190:39091)
0806 02:45:49.321090 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:49.321094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:49.322082 (+ 988us) server_negotiation.cc:407] Connection header received
0806 02:45:49.323052 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:49.323056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:49.323116 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:49.323225 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:49.324809 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.325421 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.326626 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.326849 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.326961 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:49.329517 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:49.329521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:49.329523 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:49.329548 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:49.330786 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:49.331251 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:49.331254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:49.331255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:49.331326 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:49.331619 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:49.331621 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:49.331622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:49.331785 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:49.331859 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:49.332396 (+ 537us) server_negotiation.cc:299] Negotiation successful
0806 02:45:49.332506 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:45:49.333554 11629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.321271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.1.183.131:43145)
0806 02:45:49.321945 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:49.321987 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:49.322762 (+ 775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:49.323408 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:49.323417 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:49.323934 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:49.324659 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:49.324681 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.325557 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.325560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:49.326502 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:49.326511 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.327002 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.327006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:49.327187 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:49.327743 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:49.327759 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:49.329363 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:49.330889 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:49.330894 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:49.330905 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:49.331163 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:49.331406 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:49.331408 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:49.331409 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:49.331541 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:49.331872 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:49.331877 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:49.332110 (+ 233us) client_negotiation.cc:769] Sending connection context
0806 02:45:49.332356 (+ 246us) client_negotiation.cc:240] Negotiation successful
0806 02:45:49.332623 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":130,"threads_started":1}
I20260806 02:45:49.334738 11627 heartbeater.cc:347] Connected to a master server at 127.1.183.190:39091
I20260806 02:45:49.335016 11627 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:49.335506 11627 heartbeater.cc:511] Master 127.1.183.190:39091 requested a full tablet report, sending...
I20260806 02:45:49.336704 10983 ts_manager.cc:194] Registered new tserver with Master: e2cbd8619d9048b5b56e4d9ea417288e (127.1.183.131:39727)
I20260806 02:45:49.337386 1758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:45:49.337481 10983 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.183.131@KRBTEST.COM'} at 127.1.183.131:43145
WARNING: no policy specified for oryx/127.1.183.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.183.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.183.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.189.keytab.
Entry for principal oryx/127.1.183.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.189.keytab.
I20260806 02:45:49.368276 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.189:40231
--webserver_interface=127.1.183.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.189.keytab
--principal=oryx/127.1.183.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:39639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:39091,127.1.183.189:40231
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:49.490025 11634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:49.490331 11634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:49.490374 11634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:49.494443 11634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:49.494568 11634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:49.494590 11634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:49.494608 11634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:49.494623 11634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:49.500226 11634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:39639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:39091,127.1.183.189:40231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.189.keytab
--principal=oryx/127.1.183.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.189:40231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.183.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
W20260806 02:45:49.500453 11634 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.1.183.190:39091,127.1.183.189:40231'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260806 02:45:49.501382 11634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:49.502411 11634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:49.511432 11641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:49.511443 11642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:49.511591 11634 server_base.cc:1034] running on GCE node
W20260806 02:45:49.511831 11645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:49.512506 11634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:49.513718 11634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:49.514948 11634 hybrid_clock.cc:648] HybridClock initialized: now 1785984349514911 us; error 49 us; skew 500 ppm
Aug 06 02:45:49 dist-test-slave-gmjp krb5kdc[10919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984349, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:49.518091 11634 init.cc:377] Logged in from keytab as oryx/127.1.183.189@KRBTEST.COM (short username oryx)
I20260806 02:45:49.519145 11634 webserver.cc:460] Webserver started at http://127.1.183.189:34029/ using document root <none> and password file <none>
I20260806 02:45:49.519752 11634 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:49.519846 11634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:49.520094 11634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:49.522418 11634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-1/data/instance:
uuid: "fb3e8a85dcf24fb7b146b28dd2053249"
format_stamp: "Formatted at 2026-08-06 02:45:49 on dist-test-slave-gmjp"
server_key: "6d292fbd2d719884afca56829984b871"
server_key_iv: "ec029e4a7a2331e67bd19626082b6cdf"
server_key_version: "encryptionkey@0"
I20260806 02:45:49.523038 11634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-1/wal/instance:
uuid: "fb3e8a85dcf24fb7b146b28dd2053249"
format_stamp: "Formatted at 2026-08-06 02:45:49 on dist-test-slave-gmjp"
server_key: "6d292fbd2d719884afca56829984b871"
server_key_iv: "ec029e4a7a2331e67bd19626082b6cdf"
server_key_version: "encryptionkey@0"
I20260806 02:45:49.527426 11634 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:45:49.530053 11654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:49.531311 11634 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 02:45:49.531471 11634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-1/wal
uuid: "fb3e8a85dcf24fb7b146b28dd2053249"
format_stamp: "Formatted at 2026-08-06 02:45:49 on dist-test-slave-gmjp"
server_key: "6d292fbd2d719884afca56829984b871"
server_key_iv: "ec029e4a7a2331e67bd19626082b6cdf"
server_key_version: "encryptionkey@0"
I20260806 02:45:49.531596 11634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:49.545207 11634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:49.548239 11634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:49.548447 11634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:49.556339 11634 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.189:40231
I20260806 02:45:49.556674 11726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.189:40231 every 8 connection(s)
I20260806 02:45:49.557549 11634 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-1/data/info.pb
I20260806 02:45:49.560564 11727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:49.564467 11727 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:45:49.566962 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 11634
Aug 06 02:45:49 dist-test-slave-gmjp krb5kdc[10919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984349, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.189@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM
Aug 06 02:45:49 dist-test-slave-gmjp krb5kdc[10919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.183.189@KRBTEST.COM
I20260806 02:45:49.579139 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.566389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53532 (local address 127.1.183.190:39091)
0806 02:45:49.566545 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:49.566551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:49.567634 (+ 1083us) server_negotiation.cc:407] Connection header received
0806 02:45:49.568903 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:49.568906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:49.568953 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:49.569016 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:49.570076 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.570898 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.571953 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.572227 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.572343 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:49.574974 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:49.574979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:49.574982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:49.575015 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:49.576672 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:49.577242 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:49.577244 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:49.577246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:49.577315 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:49.577695 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:49.577699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:49.577701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:49.578021 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:49.578138 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:49.578811 (+ 673us) server_negotiation.cc:299] Negotiation successful
0806 02:45:49.578961 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:45:49.580241 11734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.567869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51158 (local address 127.1.183.189:40231)
0806 02:45:49.568243 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:49.568253 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:49.568296 (+ 43us) server_negotiation.cc:407] Connection header received
0806 02:45:49.568689 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:49.568711 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:49.568957 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:49.569168 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:49.569918 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.571186 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.572019 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.572578 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.572692 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:49.575468 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:49.575478 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:49.575493 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:49.575522 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:49.577457 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:49.577943 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:49.577948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:49.577949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:49.578019 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:49.578296 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:49.578300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:49.578301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:49.578654 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:49.578843 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:49.579067 (+ 224us) server_negotiation.cc:299] Negotiation successful
0806 02:45:49.579193 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":237,"thread_start_us":107,"threads_started":1}
I20260806 02:45:49.580241 11729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.566880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.0.0.1:53532)
0806 02:45:49.567441 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:49.567486 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:49.568689 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:49.569097 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:49.569109 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:49.569559 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:49.569923 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:49.569938 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.571032 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.571034 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:49.571781 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:49.571790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.572385 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.572389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:49.572608 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:49.573243 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:49.573267 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:49.574815 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:49.576782 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:49.576789 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:49.576801 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:49.577144 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:49.577407 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:49.577411 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:49.577414 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:49.577594 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:49.578146 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:49.578153 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:49.578491 (+ 338us) client_negotiation.cc:769] Sending connection context
0806 02:45:49.578740 (+ 249us) client_negotiation.cc:240] Negotiation successful
0806 02:45:49.579108 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":22,"thread_start_us":164,"threads_started":1}
I20260806 02:45:49.582460 11727 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:45:49 dist-test-slave-gmjp krb5kdc[10919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984349, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.189@KRBTEST.COM for oryx/127.1.183.189@KRBTEST.COM
I20260806 02:45:49.590432 11729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.582993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:40231 (local address 127.0.0.1:51160)
0806 02:45:49.583129 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:49.583144 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:49.583224 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:49.583435 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:49.583439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:49.583611 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:49.583708 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:49.583713 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.584583 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.584586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:49.585092 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:49.585100 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.585526 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.585529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:49.585602 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:49.586064 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:49.586075 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:49.587292 (+ 1217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:49.589155 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:49.589158 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:49.589159 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:49.589341 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:49.589583 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:49.589585 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:49.589586 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:49.589657 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:49.589987 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:49.589989 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:49.590066 (+ 77us) client_negotiation.cc:769] Sending connection context
0806 02:45:49.590145 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 02:45:49.590251 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 02:45:49.590432 11734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.583088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51160 (local address 127.1.183.189:40231)
0806 02:45:49.583252 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:49.583256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:49.583266 (+ 10us) server_negotiation.cc:407] Connection header received
0806 02:45:49.583295 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:49.583297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:49.583328 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:49.583390 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:49.583857 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.584463 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.585201 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.585429 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.585508 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:49.587418 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:49.587421 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:49.587423 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:49.587444 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:49.589039 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:49.589432 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:49.589434 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:49.589435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:49.589505 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:49.589731 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:49.589733 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:49.589734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:49.589895 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:49.589980 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:49.590202 (+ 222us) server_negotiation.cc:299] Negotiation successful
0806 02:45:49.590304 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":98}
I20260806 02:45:49.594038 11727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249: Bootstrap starting.
I20260806 02:45:49.596447 11727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:49.597208 11727 log.cc:824] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:49.599138 11727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249: No bootstrap required, opened a new log
I20260806 02:45:49.602172 11727 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } }
I20260806 02:45:49.602363 11727 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:49.602440 11727 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb3e8a85dcf24fb7b146b28dd2053249, State: Initialized, Role: FOLLOWER
I20260806 02:45:49.602898 11727 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } }
I20260806 02:45:49.603379 11736 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } } }
I20260806 02:45:49.603518 11736 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:45:49.604110 11727 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 02:45:49.609295 11754 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:45:49.609382 11754 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:45:49.611642 11634 master_runner.cc:428] Detected that this master fb3e8a85dcf24fb7b146b28dd2053249 is joining an existing cluster
I20260806 02:45:49.611706 11634 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.183.189:40231
I20260806 02:45:49.622954 11734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.614293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51176 (local address 127.1.183.189:40231)
0806 02:45:49.614391 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:49.614395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:49.614420 (+ 25us) server_negotiation.cc:407] Connection header received
0806 02:45:49.614798 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:49.614801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:49.614841 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:49.614915 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:49.615373 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.615852 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.617018 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.617356 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.617479 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:49.619356 (+ 1877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:49.619359 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:49.619361 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:49.619383 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:49.621061 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:49.621556 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:49.621561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:49.621563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:49.621643 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:49.622005 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:49.622011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:49.622016 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:49.622207 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:49.622323 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:49.622574 (+ 251us) server_negotiation.cc:299] Negotiation successful
0806 02:45:49.622733 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":38}
I20260806 02:45:49.622989 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.613919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53540 (local address 127.1.183.190:39091)
0806 02:45:49.614029 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:49.614033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:49.614319 (+ 286us) server_negotiation.cc:407] Connection header received
0806 02:45:49.614534 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:49.614538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:49.614591 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:49.614678 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:49.615254 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.616084 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.617182 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.617445 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.617547 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:49.619162 (+ 1615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:49.619166 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:49.619169 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:49.619196 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:49.621019 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:49.621621 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:49.621625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:49.621626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:49.621731 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:49.622056 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:49.622060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:49.622062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:49.622275 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:49.622366 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:49.622740 (+ 374us) server_negotiation.cc:299] Negotiation successful
0806 02:45:49.622854 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260806 02:45:49.623019 11762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.613855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.0.0.1:53540)
0806 02:45:49.614217 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:49.614235 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:49.614415 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:49.614709 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:49.614711 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:49.614954 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:49.615122 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:49.615129 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.616218 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.616221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:49.617075 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:49.617082 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.617670 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.617676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:49.617806 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:49.618360 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:49.618380 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:49.619009 (+ 629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:49.621147 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:49.621151 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:49.621153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:49.621495 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:49.621819 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:49.621822 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:49.621824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:49.621914 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:49.622429 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:49.622432 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:49.622518 (+ 86us) client_negotiation.cc:769] Sending connection context
0806 02:45:49.622673 (+ 155us) client_negotiation.cc:240] Negotiation successful
0806 02:45:49.622810 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"spinlock_wait_cycles":2688,"thread_start_us":245,"threads_started":1}
I20260806 02:45:49.623638 11764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.613909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:40231 (local address 127.0.0.1:51176)
0806 02:45:49.614313 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:49.614324 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:49.614415 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:49.614951 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:49.614954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:49.615136 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:49.615258 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:49.615264 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.615965 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.615972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:49.616891 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:49.616901 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.617501 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.617507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:49.617593 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:49.618760 (+ 1167us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:49.618775 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:49.619216 (+ 441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:49.621174 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:49.621180 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:49.621182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:49.621458 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:49.621740 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:49.621743 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:49.621744 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:49.621887 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:49.622304 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:49.622307 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:49.622414 (+ 107us) client_negotiation.cc:769] Sending connection context
0806 02:45:49.622527 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 02:45:49.622843 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"spinlock_wait_cycles":4608,"thread_start_us":107,"threads_started":1}
Aug 06 02:45:49 dist-test-slave-gmjp krb5kdc[10919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984348, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.190@KRBTEST.COM for oryx/127.1.183.189@KRBTEST.COM
I20260806 02:45:49.640262 11734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.628642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51184 (local address 127.1.183.189:40231)
0806 02:45:49.628763 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:49.628767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:49.629501 (+ 734us) server_negotiation.cc:407] Connection header received
0806 02:45:49.629695 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:49.629697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:49.629742 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:49.629805 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:49.630608 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.631101 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.632401 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.632626 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.632835 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:49.636353 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:49.636360 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:49.636367 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:49.636402 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:49.637887 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:49.638434 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:49.638441 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:49.638446 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:49.638547 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:49.638876 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:49.638883 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:49.638887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:49.639160 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:49.639274 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:49.639807 (+ 533us) server_negotiation.cc:299] Negotiation successful
0806 02:45:49.639960 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 02:45:49.641012 11766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.628563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:40231 (local address 127.0.0.1:51184)
0806 02:45:49.629362 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:49.629424 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:49.629557 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:49.629866 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:49.629872 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:49.630325 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:49.630493 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:49.630501 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.631209 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.631213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:49.632258 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:49.632269 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.632973 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.632978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:49.633134 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:49.633900 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:49.633920 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:49.636133 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:49.638022 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:49.638035 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:49.638042 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:49.638313 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:49.638663 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:49.638666 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:49.638668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:49.638772 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:49.639342 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:49.639353 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:49.639517 (+ 164us) client_negotiation.cc:769] Sending connection context
0806 02:45:49.640438 (+ 921us) client_negotiation.cc:240] Negotiation successful
0806 02:45:49.640737 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":89,"threads_started":1}
I20260806 02:45:49.642889 10983 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.183.189:40231: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249"
member_type: NON_VOTER
last_known_addr {
host: "127.1.183.189"
port: 40231
}
attrs {
promote: true
}
}
dest_uuid: "5975de0199eb4bc1a2ac5d5607e1928c"
cas_config_opid_index: -1
I20260806 02:45:49.643750 10983 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: NON_VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.1.183.188@KRBTEST.COM; defaulting to no policy
I20260806 02:45:49.646135 11769 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER fb3e8a85dcf24fb7b146b28dd2053249 (127.1.183.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: NON_VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: true } } }
I20260806 02:45:49.647100 11693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:49.648031 11769 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.183.189:40231
I20260806 02:45:49.648424 11771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: NON_VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: true } } }
I20260806 02:45:49.648626 11771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:49.648690 11693 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [term 1 FOLLOWER]: Refusing update from remote peer 5975de0199eb4bc1a2ac5d5607e1928c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260806 02:45:49.648788 11634 master.cc:537] Master@127.1.183.189:40231 shutting down...
I20260806 02:45:49.649529 11771 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: NON_VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.183.188@KRBTEST.COM" created.
W20260806 02:45:49.652462 10964 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260806 02:45:49.653493 10964 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c -> Peer fb3e8a85dcf24fb7b146b28dd2053249 (127.1.183.189:40231): Couldn't send request to peer fb3e8a85dcf24fb7b146b28dd2053249. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 02:45:49.654830 11634 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:49.655145 11634 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:49.655294 11634 tablet_replica.cc:330] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.183.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.188.keytab.
Entry for principal oryx/127.1.183.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.188.keytab.
I20260806 02:45:49.671139 11634 master.cc:552] Master@127.1.183.189:40231 shutdown complete.
I20260806 02:45:49.671204 11634 master_runner.cc:305] Clearing existing system tablet
I20260806 02:45:49.671159 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.188:35427
--webserver_interface=127.1.183.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.188.keytab
--principal=oryx/127.1.183.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:39639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:39091,127.1.183.189:40231,127.1.183.188:35427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:45:49.671463 11634 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:49.671548 11634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:49.673049 11634 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 02:45:49.674036 11782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:49.674283 11634 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:49.674420 11634 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 02:45:49.674453 11634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-1/wal
uuid: "fb3e8a85dcf24fb7b146b28dd2053249"
format_stamp: "Formatted at 2026-08-06 02:45:49 on dist-test-slave-gmjp"
server_key: "6d292fbd2d719884afca56829984b871"
server_key_iv: "ec029e4a7a2331e67bd19626082b6cdf"
server_key_version: "encryptionkey@0"
I20260806 02:45:49.676779 11634 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:49.679250 11634 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 02:45:49.679349 11634 log.cc:1196] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260806 02:45:49.679673 11634 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249: Deleting consensus metadata
I20260806 02:45:49.679847 11634 master_runner.cc:315] Copying system tablet from 127.1.183.190:39091
I20260806 02:45:49.681144 11634 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249: tablet copy: Beginning tablet copy session from remote peer at address 127.1.183.190:39091
I20260806 02:45:49.691330 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.681879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53542 (local address 127.1.183.190:39091)
0806 02:45:49.682011 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:49.682016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:49.682470 (+ 454us) server_negotiation.cc:407] Connection header received
0806 02:45:49.682642 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:49.682645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:49.682698 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:49.682795 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:49.683489 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.684380 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.685650 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.685996 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.686105 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:49.687606 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:49.687610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:49.687614 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:49.687645 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:49.689431 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:49.689938 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:49.689942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:49.689944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:49.690036 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:49.690335 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:49.690338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:49.690340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:49.690606 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:49.690728 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:49.691037 (+ 309us) server_negotiation.cc:299] Negotiation successful
0806 02:45:49.691166 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:45:49.691344 11788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.681797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.0.0.1:53542)
0806 02:45:49.682350 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:49.682367 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:49.682517 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:49.682850 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:49.682856 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:49.683138 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:49.683327 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:49.683340 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.684521 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.684525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:49.685510 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:49.685521 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.686247 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.686251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:49.686351 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:49.686970 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:49.686986 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:49.687441 (+ 455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:49.689555 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:49.689559 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:49.689561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:49.689833 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:49.690136 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:49.690140 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:49.690142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:49.690240 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:49.690722 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:49.690725 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:49.690834 (+ 109us) client_negotiation.cc:769] Sending connection context
0806 02:45:49.690944 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 02:45:49.691115 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":166,"threads_started":1}
I20260806 02:45:49.692387 11007 tablet_copy_service.cc:140] P 5975de0199eb4bc1a2ac5d5607e1928c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fb3e8a85dcf24fb7b146b28dd2053249 ({username='oryx', principal='oryx/127.1.183.189@KRBTEST.COM'} at 127.0.0.1:53542)
I20260806 02:45:49.692603 11007 tablet_copy_service.cc:161] P 5975de0199eb4bc1a2ac5d5607e1928c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fb3e8a85dcf24fb7b146b28dd2053249 at {username='oryx', principal='oryx/127.1.183.189@KRBTEST.COM'} at 127.0.0.1:53542: session id = fb3e8a85dcf24fb7b146b28dd2053249-00000000000000000000000000000000
I20260806 02:45:49.694550 11007 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 02:45:49.696482 11634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 02:45:49.699064 11634 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249: tablet copy: Starting download of 0 data blocks...
I20260806 02:45:49.699277 11634 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249: tablet copy: Starting download of 1 WAL segments...
I20260806 02:45:49.701297 11634 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 02:45:49.703366 11007 tablet_copy_service.cc:342] P 5975de0199eb4bc1a2ac5d5607e1928c: Request end of tablet copy session fb3e8a85dcf24fb7b146b28dd2053249-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.183.189@KRBTEST.COM'} at 127.0.0.1:53542
I20260806 02:45:49.703517 11007 tablet_copy_service.cc:433] P 5975de0199eb4bc1a2ac5d5607e1928c: ending tablet copy session fb3e8a85dcf24fb7b146b28dd2053249-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fb3e8a85dcf24fb7b146b28dd2053249
I20260806 02:45:49.711593 11634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:49.713727 11794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:49.713745 11789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:49.714061 11790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:49.720398 11634 server_base.cc:1034] running on GCE node
I20260806 02:45:49.720613 11634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:49.720989 11634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:49.722159 11634 hybrid_clock.cc:648] HybridClock initialized: now 1785984349722134 us; error 45 us; skew 500 ppm
Aug 06 02:45:49 dist-test-slave-gmjp krb5kdc[10919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984349, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:49.723929 11634 init.cc:377] Logged in from keytab as oryx/127.1.183.189@KRBTEST.COM (short username oryx)
I20260806 02:45:49.724499 11634 webserver.cc:460] Webserver started at http://127.1.183.189:38515/ using document root <none> and password file <none>
I20260806 02:45:49.724706 11634 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:49.724756 11634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:49.726368 11634 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 02:45:49.727486 11804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:49.727789 11634 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 02:45:49.727869 11634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-1/wal
uuid: "fb3e8a85dcf24fb7b146b28dd2053249"
format_stamp: "Formatted at 2026-08-06 02:45:49 on dist-test-slave-gmjp"
server_key: "6d292fbd2d719884afca56829984b871"
server_key_iv: "ec029e4a7a2331e67bd19626082b6cdf"
server_key_version: "encryptionkey@0"
I20260806 02:45:49.727953 11634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:49.750064 11634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:49.752506 11634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:49.758666 11634 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.189:40231
I20260806 02:45:49.758788 11879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.189:40231 every 8 connection(s)
I20260806 02:45:49.759317 11634 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-1/data/info.pb
I20260806 02:45:49.761467 11880 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:45:49.762665 11880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249: Bootstrap starting.
I20260806 02:45:49.774349 11880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:45:49.774751 11880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249: Bootstrap complete.
I20260806 02:45:49.775493 11880 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: NON_VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: true } }
I20260806 02:45:49.775667 11880 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fb3e8a85dcf24fb7b146b28dd2053249, State: Initialized, Role: LEARNER
I20260806 02:45:49.775815 11880 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: NON_VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: true } }
I20260806 02:45:49.776254 11887 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: NON_VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: true } } }
I20260806 02:45:49.776391 11880 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:49.776495 11887 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [sys.catalog]: This master's current role is: LEARNER
I20260806 02:45:49.780375 11902 catalog_manager.cc:1259] Loaded cluster ID: 07ac2c30b95046aa9ad9b2c2891f35ca
I20260806 02:45:49.780465 11902 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249: loading cluster ID for follower catalog manager: success
I20260806 02:45:49.782941 11902 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249: acquiring CA information for follower catalog manager: success
I20260806 02:45:49.784586 11902 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 02:45:49.812278 11777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:49.812613 11777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:49.812721 11777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:49.816943 11777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:49.817036 11777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:49.817072 11777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:49.817130 11777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:49.817169 11777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:49.822870 11777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:39639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:39091,127.1.183.189:40231,127.1.183.188:35427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.188.keytab
--principal=oryx/127.1.183.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.188:35427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.183.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:49.824028 11777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:49.825219 11777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:49.834400 11915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:49.834406 11912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:49.834400 11911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:49.834978 11777 server_base.cc:1034] running on GCE node
I20260806 02:45:49.835565 11777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:49.836905 11777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:49.838095 11777 hybrid_clock.cc:648] HybridClock initialized: now 1785984349838064 us; error 51 us; skew 500 ppm
Aug 06 02:45:49 dist-test-slave-gmjp krb5kdc[10919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984349, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:49.841024 11777 init.cc:377] Logged in from keytab as oryx/127.1.183.188@KRBTEST.COM (short username oryx)
I20260806 02:45:49.841970 11777 webserver.cc:460] Webserver started at http://127.1.183.188:41851/ using document root <none> and password file <none>
I20260806 02:45:49.842555 11777 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:49.842650 11777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:49.842891 11777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:49.845093 11777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-2/data/instance:
uuid: "938ea1eb1f524f2eacb24a0224ce4694"
format_stamp: "Formatted at 2026-08-06 02:45:49 on dist-test-slave-gmjp"
server_key: "06c5be7315b61ffaba6d6a1a013c38bd"
server_key_iv: "d507bb9050a3d31bbd03350f15d2ddc8"
server_key_version: "encryptionkey@0"
I20260806 02:45:49.845697 11777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-2/wal/instance:
uuid: "938ea1eb1f524f2eacb24a0224ce4694"
format_stamp: "Formatted at 2026-08-06 02:45:49 on dist-test-slave-gmjp"
server_key: "06c5be7315b61ffaba6d6a1a013c38bd"
server_key_iv: "d507bb9050a3d31bbd03350f15d2ddc8"
server_key_version: "encryptionkey@0"
I20260806 02:45:49.849715 11777 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:45:49.852376 11924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:49.853770 11777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:49.853950 11777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-2/wal
uuid: "938ea1eb1f524f2eacb24a0224ce4694"
format_stamp: "Formatted at 2026-08-06 02:45:49 on dist-test-slave-gmjp"
server_key: "06c5be7315b61ffaba6d6a1a013c38bd"
server_key_iv: "d507bb9050a3d31bbd03350f15d2ddc8"
server_key_version: "encryptionkey@0"
I20260806 02:45:49.854085 11777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:49.855998 11262 heartbeater.cc:503] Master 127.1.183.190:39091 was elected leader, sending a full tablet report...
I20260806 02:45:49.875162 11777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:49.879215 11777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:49.879453 11777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:49.888315 11777 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.188:35427
I20260806 02:45:49.888630 11991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.188:35427 every 8 connection(s)
I20260806 02:45:49.889523 11777 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-2/data/info.pb
I20260806 02:45:49.892477 11992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:49.896462 11992 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:45:49.899596 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 11777
Aug 06 02:45:49 dist-test-slave-gmjp krb5kdc[10919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984349, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.188@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM
Aug 06 02:45:49 dist-test-slave-gmjp krb5kdc[10919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.183.188@KRBTEST.COM
I20260806 02:45:49.911374 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.898519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53550 (local address 127.1.183.190:39091)
0806 02:45:49.898675 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:49.898679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:49.899445 (+ 766us) server_negotiation.cc:407] Connection header received
0806 02:45:49.900555 (+ 1110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:49.900559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:49.900624 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:49.900717 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:49.901950 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.902593 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.903998 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.904374 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.904498 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:49.907325 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:49.907330 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:49.907333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:49.907367 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:49.909187 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:49.909761 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:49.909764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:49.909766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:49.909861 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:49.910215 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:49.910218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:49.910220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:49.910418 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:49.910535 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:49.911054 (+ 519us) server_negotiation.cc:299] Negotiation successful
0806 02:45:49.911179 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 02:45:49.912298 11996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.900571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37704 (local address 127.1.183.188:35427)
0806 02:45:49.900996 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:49.901015 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:49.901035 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:45:49.901069 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:49.901073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:49.901221 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:49.901391 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:49.902244 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.903477 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.904523 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.905035 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.905128 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:49.907949 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:49.907958 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:49.907973 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:49.908002 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:49.909944 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:49.910351 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:49.910357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:49.910359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:49.910447 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:49.910733 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:49.910737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:49.910739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:49.910975 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:49.911236 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:49.911374 (+ 138us) server_negotiation.cc:299] Negotiation successful
0806 02:45:49.911529 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":300,"thread_start_us":167,"threads_started":1}
I20260806 02:45:49.912307 11995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.898804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.0.0.1:53550)
0806 02:45:49.899270 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:49.899331 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:49.900313 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:49.900938 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:49.900945 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:49.901285 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:49.901789 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:49.901802 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.902736 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.902739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:49.903883 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:49.903892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.904540 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.904544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:49.904735 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:49.905402 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:49.905417 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:49.907175 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:49.909308 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:49.909316 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:49.909327 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:49.909652 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:49.909971 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:49.909973 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:49.909975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:49.910108 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:49.910533 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:49.910540 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:49.910787 (+ 247us) client_negotiation.cc:769] Sending connection context
0806 02:45:49.911008 (+ 221us) client_negotiation.cc:240] Negotiation successful
0806 02:45:49.911321 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":111,"thread_start_us":121,"threads_started":1}
I20260806 02:45:49.913647 11992 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:45:49 dist-test-slave-gmjp krb5kdc[10919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984349, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.188@KRBTEST.COM for oryx/127.1.183.189@KRBTEST.COM
I20260806 02:45:49.922065 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.914281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51200 (local address 127.1.183.189:40231)
0806 02:45:49.914588 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:49.914592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:49.914606 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:45:49.914651 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:49.914654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:49.914705 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:49.914837 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:49.915219 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.915923 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.916756 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.917003 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.917117 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:49.919039 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:49.919042 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:49.919044 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:49.919067 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:49.920423 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:49.920827 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:49.920831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:49.920833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:49.920930 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:49.921214 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:49.921218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:49.921220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:49.921416 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:49.921529 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:49.921786 (+ 257us) server_negotiation.cc:299] Negotiation successful
0806 02:45:49.921910 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":100,"threads_started":1}
I20260806 02:45:49.922056 11995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.914145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:40231 (local address 127.0.0.1:51200)
0806 02:45:49.914265 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:49.914281 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:49.914360 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:49.914831 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:49.914834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:49.915008 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:49.915097 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:49.915103 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.916035 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.916041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:49.916634 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:49.916642 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.917116 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.917119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:49.917221 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:49.917707 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:49.917718 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:49.918905 (+ 1187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:49.920523 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:49.920526 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:49.920528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:49.920717 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:49.921030 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:49.921034 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:49.921036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:49.921117 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:49.921520 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:49.921523 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:49.921618 (+ 95us) client_negotiation.cc:769] Sending connection context
0806 02:45:49.921727 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 02:45:49.921880 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 02:45:49.922935 11992 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35427 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:45:49 dist-test-slave-gmjp krb5kdc[10919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984349, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.188@KRBTEST.COM for oryx/127.1.183.188@KRBTEST.COM
I20260806 02:45:49.930573 11995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.923459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:35427 (local address 127.0.0.1:37720)
0806 02:45:49.923568 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:49.923581 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:49.923642 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:49.923862 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:49.923864 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:49.923989 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:49.924057 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:49.924062 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.924795 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.924798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:49.925337 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:49.925346 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.925911 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.925914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:49.925974 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:49.926492 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:49.926504 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:49.927753 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:49.929302 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:49.929305 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:49.929306 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:49.929497 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:49.929731 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:49.929733 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:49.929734 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:49.929805 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:49.930140 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:49.930142 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:49.930214 (+ 72us) client_negotiation.cc:769] Sending connection context
0806 02:45:49.930292 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 02:45:49.930406 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 02:45:49.930588 11996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.923571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37720 (local address 127.1.183.188:35427)
0806 02:45:49.923683 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:49.923686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:49.923697 (+ 11us) server_negotiation.cc:407] Connection header received
0806 02:45:49.923724 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:49.923726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:49.923756 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:49.923817 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:49.924182 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.924685 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.925437 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.925813 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.925891 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:49.927874 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:49.927877 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:49.927879 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:49.927899 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:49.929199 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:49.929578 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:49.929580 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:49.929581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:49.929650 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:49.929881 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:49.929883 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:49.929884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:49.930061 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:49.930134 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:49.930346 (+ 212us) server_negotiation.cc:299] Negotiation successful
0806 02:45:49.930461 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":41}
I20260806 02:45:49.934185 11992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694: Bootstrap starting.
I20260806 02:45:49.936741 11992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:49.937508 11992 log.cc:824] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:49.939535 11992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694: No bootstrap required, opened a new log
I20260806 02:45:49.942703 11992 raft_consensus.cc:348] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35427 } }
I20260806 02:45:49.942941 11992 raft_consensus.cc:374] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:49.943022 11992 raft_consensus.cc:729] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 938ea1eb1f524f2eacb24a0224ce4694, State: Initialized, Role: FOLLOWER
I20260806 02:45:49.943480 11992 consensus_queue.cc:260] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35427 } }
I20260806 02:45:49.944075 11999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35427 } } }
I20260806 02:45:49.944290 11999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:45:49.944931 11992 sys_catalog.cc:564] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 02:45:49.950400 12015 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:45:49.950508 12015 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:45:49.954188 11777 master_runner.cc:428] Detected that this master 938ea1eb1f524f2eacb24a0224ce4694 is joining an existing cluster
I20260806 02:45:49.954267 11777 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.183.188:35427
I20260806 02:45:49.965916 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 11067
I20260806 02:45:49.981238 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.129:34391
--local_ip_for_outbound_sockets=127.1.183.129
--tserver_master_addrs=127.1.183.190:39091
--webserver_port=45685
--webserver_interface=127.1.183.129
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.129.keytab
--principal=oryx/127.1.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:39639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.183.190:39091,127.1.183.189:40231
--tserver_master_addrs=127.1.183.190:39091,127.1.183.189:40231,127.1.183.188:35427 with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:45:49.990425 12024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.956465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.0.0.1:53552)
0806 02:45:49.957373 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:49.957387 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:49.957514 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:49.957908 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:49.957910 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:49.958206 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:49.958398 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:49.958407 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.959934 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.959939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:49.960944 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:49.960955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.961687 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.961691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:49.961790 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:49.984845 (+ 23055us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:49.984868 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:49.985397 (+ 529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:49.988495 (+ 3098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:49.988499 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:49.988502 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:49.988765 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:49.989117 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:49.989121 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:49.989123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:49.989216 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:49.989751 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:49.989754 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:49.989892 (+ 138us) client_negotiation.cc:769] Sending connection context
0806 02:45:49.989998 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 02:45:49.990184 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"mutex_wait_us":27,"spinlock_wait_cycles":32896,"thread_start_us":120,"threads_started":1}
I20260806 02:45:49.991163 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.956424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53552 (local address 127.1.183.190:39091)
0806 02:45:49.956676 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:49.956681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:49.957479 (+ 798us) server_negotiation.cc:407] Connection header received
0806 02:45:49.957640 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:49.957645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:49.957702 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:49.957778 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:49.958556 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.959398 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.961094 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.961456 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.961562 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:49.985608 (+ 24046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:49.985613 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:49.985615 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:49.985648 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:49.988344 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:49.988903 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:49.988907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:49.988909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:49.989002 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:49.989337 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:49.989340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:49.989343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:49.989603 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:49.989704 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:49.990839 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0806 02:45:49.990976 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260806 02:45:49.992061 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.956517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51204 (local address 127.1.183.189:40231)
0806 02:45:49.956671 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:49.956676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:49.957522 (+ 846us) server_negotiation.cc:407] Connection header received
0806 02:45:49.957600 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:49.957603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:49.957660 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:49.957744 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:49.958865 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.959720 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.961452 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.961692 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.961807 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:49.986047 (+ 24240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:49.986052 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:49.986056 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:49.986093 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:49.988762 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:49.989294 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:49.989300 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:49.989303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:49.989402 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:49.990095 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:49.990098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:49.990100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:49.990317 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:49.990391 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:49.991781 (+ 1390us) server_negotiation.cc:299] Negotiation successful
0806 02:45:49.991922 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:45:49.992834 12029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.956797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:40231 (local address 127.0.0.1:51204)
0806 02:45:49.957372 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:49.957387 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:49.957504 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:49.957877 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:49.957880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:49.958610 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:49.958727 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:49.958733 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.960412 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.960415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:49.961329 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:49.961339 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.961811 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.961814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:49.961916 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:49.962421 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:49.962431 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:49.985397 (+ 22966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:49.988903 (+ 3506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:49.988906 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:49.988908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:49.989166 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:49.989885 (+ 719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:49.989888 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:49.989890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:49.989993 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:49.990474 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:49.990479 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:49.990911 (+ 432us) client_negotiation.cc:769] Sending connection context
0806 02:45:49.992484 (+ 1573us) client_negotiation.cc:240] Negotiation successful
0806 02:45:49.992643 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":1,"spinlock_wait_cycles":7552,"thread_start_us":399,"threads_started":1}
I20260806 02:45:49.994047 11996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.956974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37724 (local address 127.1.183.188:35427)
0806 02:45:49.957110 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:49.957117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:49.957854 (+ 737us) server_negotiation.cc:407] Connection header received
0806 02:45:49.958006 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:49.958010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:49.958046 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:49.958094 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:49.958622 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.959397 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.960351 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.960720 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.960819 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:49.985596 (+ 24777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:49.985600 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:49.985603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:49.985627 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:49.987874 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:49.991070 (+ 3196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:49.991073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:49.991075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:49.991163 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:49.993130 (+ 1967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:49.993134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:49.993136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:49.993332 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:49.993418 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:49.993715 (+ 297us) server_negotiation.cc:299] Negotiation successful
0806 02:45:49.993855 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:45:49.994457 12023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.956956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:35427 (local address 127.0.0.1:37724)
0806 02:45:49.957786 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:49.957799 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:49.957903 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:49.958215 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:49.958217 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:49.958384 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:49.958514 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:49.958521 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.959591 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.959594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:49.960236 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:49.960245 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.960956 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.960959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:49.961042 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:49.961975 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:49.961997 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:49.984897 (+ 22900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:49.989502 (+ 4605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:49.989505 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:49.989507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:49.989779 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:49.991515 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:49.991518 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:49.991519 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:49.991614 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:49.993464 (+ 1850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:49.993466 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:49.993542 (+ 76us) client_negotiation.cc:769] Sending connection context
0806 02:45:49.994090 (+ 548us) client_negotiation.cc:240] Negotiation successful
0806 02:45:49.994249 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"spinlock_wait_cycles":1920,"thread_start_us":159,"threads_started":1}
Aug 06 02:45:49 dist-test-slave-gmjp krb5kdc[10919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984348, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.190@KRBTEST.COM for oryx/127.1.183.188@KRBTEST.COM
I20260806 02:45:50.003746 11766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.995660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:35427 (local address 127.0.0.1:37738)
0806 02:45:49.995801 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:49.995832 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:49.995922 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:49.996376 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:49.996379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:49.996542 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:49.996627 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:49.996632 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.997404 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.997409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:49.997974 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:49.997983 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.998447 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.998462 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:49.998558 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:49.999023 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:49.999037 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:50.000345 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:50.002214 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:50.002217 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:50.002219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:50.002436 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:50.002726 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:50.002729 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:50.002731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:50.002821 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:50.003192 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:50.003195 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:50.003313 (+ 118us) client_negotiation.cc:769] Sending connection context
0806 02:45:50.003421 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 02:45:50.003583 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 02:45:50.003746 11996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:49.995897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37738 (local address 127.1.183.188:35427)
0806 02:45:49.996006 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:49.996013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:49.996027 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:45:49.996091 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:49.996094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:49.996143 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:49.996249 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:49.996755 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.997262 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.998109 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:49.998331 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:49.998442 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.000479 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:50.000482 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:50.000485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:50.000514 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:50.002075 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:50.002543 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:50.002547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:50.002549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:50.002628 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:50.002919 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:50.002922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:50.002924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:50.003091 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:50.003201 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:50.003489 (+ 288us) server_negotiation.cc:299] Negotiation successful
0806 02:45:50.003596 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 02:45:50.005189 10983 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.183.188:35427: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694"
member_type: NON_VOTER
last_known_addr {
host: "127.1.183.188"
port: 35427
}
attrs {
promote: true
}
}
dest_uuid: "5975de0199eb4bc1a2ac5d5607e1928c"
cas_config_opid_index: 5
I20260806 02:45:50.005791 10983 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: NON_VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: true } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: NON_VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: true } }
I20260806 02:45:50.006973 11771 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 938ea1eb1f524f2eacb24a0224ce4694 (127.1.183.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: NON_VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: true } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: NON_VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: true } } }
I20260806 02:45:50.008672 11771 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.183.188:35427
I20260806 02:45:50.008880 11771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: NON_VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: true } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: NON_VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: true } } }
I20260806 02:45:50.009002 11771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:50.009670 11777 master.cc:537] Master@127.1.183.188:35427 shutting down...
I20260806 02:45:50.010035 11958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:50.011691 11958 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [term 1 FOLLOWER]: Refusing update from remote peer 5975de0199eb4bc1a2ac5d5607e1928c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 02:45:50.013157 11769 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [LEADER]: Connected to new peer: Peer: permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: NON_VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 02:45:50.015667 10966 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c -> Peer 938ea1eb1f524f2eacb24a0224ce4694 (127.1.183.188:35427): Couldn't send request to peer 938ea1eb1f524f2eacb24a0224ce4694. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 02:45:50.017957 11777 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:50.018141 11766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.007108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:40231 (local address 127.0.0.1:51212)
0806 02:45:50.010214 (+ 3106us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:50.010232 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:50.010344 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:50.010819 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:50.010822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:50.010842 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:50.010936 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.010942 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.012008 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.012012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.014810 (+ 2798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.014820 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.017369 (+ 2549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.017373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.017621 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.017801 (+ 180us) client_negotiation.cc:769] Sending connection context
0806 02:45:50.017891 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 02:45:50.017992 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3013}
I20260806 02:45:50.018584 11777 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:50.018635 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.007352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51212 (local address 127.1.183.189:40231)
0806 02:45:50.007567 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:50.007572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:50.010439 (+ 2867us) server_negotiation.cc:407] Connection header received
0806 02:45:50.010497 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:50.010500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:50.010569 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:50.010680 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:50.011069 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.011860 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.014967 (+ 3107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.016971 (+ 2004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.017088 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.017289 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:50.018385 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0806 02:45:50.018510 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260806 02:45:50.018904 11777 tablet_replica.cc:330] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694: stopping tablet replica
I20260806 02:45:50.019146 11843 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [term 1 LEARNER]: Refusing update from remote peer 5975de0199eb4bc1a2ac5d5607e1928c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 02:45:50.019657 11769 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: NON_VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 02:45:50.021734 11843 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 938ea1eb1f524f2eacb24a0224ce4694 (127.1.183.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: NON_VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: true } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: NON_VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: true } } }
I20260806 02:45:50.024053 11887 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5975de0199eb4bc1a2ac5d5607e1928c. Latest consensus state: current_term: 1 leader_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: NON_VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: true } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: NON_VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: true } } }
I20260806 02:45:50.024209 11887 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [sys.catalog]: This master's current role is: LEARNER
I20260806 02:45:50.024336 11887 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: NON_VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: true } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: NON_VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: true } } }
I20260806 02:45:50.024428 11887 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [sys.catalog]: This master's current role is: LEARNER
I20260806 02:45:50.025288 11769 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c: attempting to promote NON_VOTER fb3e8a85dcf24fb7b146b28dd2053249 to VOTER
I20260806 02:45:50.025835 11769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: false } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: NON_VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: true } }
I20260806 02:45:50.028066 11843 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [term 1 LEARNER]: Refusing update from remote peer 5975de0199eb4bc1a2ac5d5607e1928c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260806 02:45:50.028293 10966 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c -> Peer 938ea1eb1f524f2eacb24a0224ce4694 (127.1.183.188:35427): Couldn't send request to peer 938ea1eb1f524f2eacb24a0224ce4694. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 02:45:50.029464 11771 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 02:45:50.031771 11769 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, fb3e8a85dcf24fb7b146b28dd2053249 (127.1.183.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: false } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: NON_VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: true } } }
I20260806 02:45:50.032497 11843 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, fb3e8a85dcf24fb7b146b28dd2053249 (127.1.183.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: false } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: NON_VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: true } } }
I20260806 02:45:50.033632 11769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: false } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: NON_VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: true } } }
I20260806 02:45:50.033740 11769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:50.033905 11771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: false } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: NON_VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: true } } }
I20260806 02:45:50.034025 11771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:50.034294 12032 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: false } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: NON_VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: true } } }
I20260806 02:45:50.034441 12032 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:45:50.041533 11777 master.cc:552] Master@127.1.183.188:35427 shutdown complete.
I20260806 02:45:50.041620 11777 master_runner.cc:305] Clearing existing system tablet
I20260806 02:45:50.041865 11777 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:50.041949 11777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:50.043123 11777 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 02:45:50.044018 12038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:50.044303 11777 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:50.044443 11777 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 02:45:50.044487 11777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-2/wal
uuid: "938ea1eb1f524f2eacb24a0224ce4694"
format_stamp: "Formatted at 2026-08-06 02:45:49 on dist-test-slave-gmjp"
server_key: "06c5be7315b61ffaba6d6a1a013c38bd"
server_key_iv: "d507bb9050a3d31bbd03350f15d2ddc8"
server_key_version: "encryptionkey@0"
I20260806 02:45:50.046669 11777 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:50.049088 11777 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 02:45:50.049185 11777 log.cc:1196] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260806 02:45:50.049459 11777 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694: Deleting consensus metadata
I20260806 02:45:50.049649 11777 master_runner.cc:315] Copying system tablet from 127.1.183.190:39091
I20260806 02:45:50.050832 11777 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694: tablet copy: Beginning tablet copy session from remote peer at address 127.1.183.190:39091
I20260806 02:45:50.059656 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.051554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53556 (local address 127.1.183.190:39091)
0806 02:45:50.051695 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:50.051700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:50.051962 (+ 262us) server_negotiation.cc:407] Connection header received
0806 02:45:50.052101 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:50.052104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:50.052211 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:50.052295 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:50.052773 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.053327 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.054246 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.054545 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.054659 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.055981 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:50.055984 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:50.055987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:50.056021 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:50.057838 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:50.058284 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:50.058288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:50.058290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:50.058381 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:50.058682 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:50.058686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:50.058688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:50.058890 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:50.059010 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:50.059355 (+ 345us) server_negotiation.cc:299] Negotiation successful
0806 02:45:50.059476 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:45:50.059669 12045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.051508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.0.0.1:53556)
0806 02:45:50.051849 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:50.051862 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:50.052009 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:50.052311 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:50.052314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:50.052536 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:50.052644 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.052650 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.053452 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.053456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.054109 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.054118 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.054665 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.054669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.054766 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.055388 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:50.055416 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:50.055833 (+ 417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:50.057960 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:50.057965 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:50.057968 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:50.058183 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:50.058489 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:50.058497 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:50.058499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:50.058589 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:50.059001 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:50.059004 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:50.059153 (+ 149us) client_negotiation.cc:769] Sending connection context
0806 02:45:50.059292 (+ 139us) client_negotiation.cc:240] Negotiation successful
0806 02:45:50.059463 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":92,"threads_started":1}
I20260806 02:45:50.060598 11007 tablet_copy_service.cc:140] P 5975de0199eb4bc1a2ac5d5607e1928c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 938ea1eb1f524f2eacb24a0224ce4694 ({username='oryx', principal='oryx/127.1.183.188@KRBTEST.COM'} at 127.0.0.1:53556)
I20260806 02:45:50.060737 11007 tablet_copy_service.cc:161] P 5975de0199eb4bc1a2ac5d5607e1928c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 938ea1eb1f524f2eacb24a0224ce4694 at {username='oryx', principal='oryx/127.1.183.188@KRBTEST.COM'} at 127.0.0.1:53556: session id = 938ea1eb1f524f2eacb24a0224ce4694-00000000000000000000000000000000
I20260806 02:45:50.061630 11007 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 02:45:50.062896 11777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 02:45:50.064988 11777 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694: tablet copy: Starting download of 0 data blocks...
I20260806 02:45:50.065209 11777 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694: tablet copy: Starting download of 1 WAL segments...
I20260806 02:45:50.067231 11777 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 02:45:50.069072 11007 tablet_copy_service.cc:342] P 5975de0199eb4bc1a2ac5d5607e1928c: Request end of tablet copy session 938ea1eb1f524f2eacb24a0224ce4694-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.183.188@KRBTEST.COM'} at 127.0.0.1:53556
I20260806 02:45:50.069181 11007 tablet_copy_service.cc:433] P 5975de0199eb4bc1a2ac5d5607e1928c: ending tablet copy session 938ea1eb1f524f2eacb24a0224ce4694-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 938ea1eb1f524f2eacb24a0224ce4694
I20260806 02:45:50.076445 11777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:50.078337 12053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:50.078383 12046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:50.079015 11777 server_base.cc:1034] running on GCE node
W20260806 02:45:50.080302 12047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:50.080551 11777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:50.080924 11777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:50.088359 11777 hybrid_clock.cc:648] HybridClock initialized: now 1785984350088338 us; error 37 us; skew 500 ppm
Aug 06 02:45:50 dist-test-slave-gmjp krb5kdc[10919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984350, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:50.090029 11777 init.cc:377] Logged in from keytab as oryx/127.1.183.188@KRBTEST.COM (short username oryx)
I20260806 02:45:50.090638 11777 webserver.cc:460] Webserver started at http://127.1.183.188:38105/ using document root <none> and password file <none>
I20260806 02:45:50.090863 11777 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:50.090919 11777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:50.092118 11777 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 02:45:50.093144 12062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:50.093379 11777 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 02:45:50.093483 11777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-2/wal
uuid: "938ea1eb1f524f2eacb24a0224ce4694"
format_stamp: "Formatted at 2026-08-06 02:45:49 on dist-test-slave-gmjp"
server_key: "06c5be7315b61ffaba6d6a1a013c38bd"
server_key_iv: "d507bb9050a3d31bbd03350f15d2ddc8"
server_key_version: "encryptionkey@0"
I20260806 02:45:50.093592 11777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:50.105221 11437 heartbeater.cc:503] Master 127.1.183.190:39091 was elected leader, sending a full tablet report...
I20260806 02:45:50.120091 11777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:50.122460 11777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:50.127908 11777 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.188:35427
I20260806 02:45:50.127933 12139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.188:35427 every 8 connection(s)
I20260806 02:45:50.128574 11777 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/master-2/data/info.pb
I20260806 02:45:50.130229 12140 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:45:50.131131 12140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694: Bootstrap starting.
I20260806 02:45:50.140859 12140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:45:50.141204 12140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694: Bootstrap complete.
I20260806 02:45:50.141757 12140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: false } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: NON_VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: true } }
I20260806 02:45:50.141907 12140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 938ea1eb1f524f2eacb24a0224ce4694, State: Initialized, Role: LEARNER
I20260806 02:45:50.142031 12140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: false } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: NON_VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: true } }
I20260806 02:45:50.142421 12145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: false } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: NON_VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: true } } }
I20260806 02:45:50.142598 12145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [sys.catalog]: This master's current role is: LEARNER
I20260806 02:45:50.142632 12140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:50.145926 12157 catalog_manager.cc:1259] Loaded cluster ID: 07ac2c30b95046aa9ad9b2c2891f35ca
I20260806 02:45:50.146028 12157 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694: loading cluster ID for follower catalog manager: success
I20260806 02:45:50.148114 12157 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694: acquiring CA information for follower catalog manager: success
I20260806 02:45:50.149303 12157 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 02:45:50.154728 12030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:50.155097 12030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:50.155164 12030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:50.161710 12030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:50.161806 12030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:50.161911 12030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.129
I20260806 02:45:50.170488 12030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:39639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.129.keytab
--principal=oryx/127.1.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.129:34391
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.183.129
--webserver_port=45685
--tserver_master_addrs=127.1.183.190:39091,127.1.183.189:40231,127.1.183.188:35427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.129
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:50.171944 12030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:50.173477 12030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:50.185600 12164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:50.185634 12169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:50.185882 12030 server_base.cc:1034] running on GCE node
W20260806 02:45:50.185603 12166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:50.186453 12030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:50.187206 12030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:50.188426 12030 hybrid_clock.cc:648] HybridClock initialized: now 1785984350188408 us; error 27 us; skew 500 ppm
Aug 06 02:45:50 dist-test-slave-gmjp krb5kdc[10919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984350, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:50.191469 12030 init.cc:377] Logged in from keytab as oryx/127.1.183.129@KRBTEST.COM (short username oryx)
I20260806 02:45:50.192458 12030 webserver.cc:460] Webserver started at http://127.1.183.129:45685/ using document root <none> and password file <none>
I20260806 02:45:50.193025 12030 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:50.193106 12030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:50.197432 12030 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:45:50.199568 12178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:50.200942 12030 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:45:50.201073 12030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-0/wal
uuid: "fff12eae637c41ebabb8689441a6dbe5"
format_stamp: "Formatted at 2026-08-06 02:45:48 on dist-test-slave-gmjp"
server_key: "7c6bca1819a9a888efb1fa1f4ce94286"
server_key_iv: "fd43a0fa4ce5cb0f32286dbdcfe33003"
server_key_version: "encryptionkey@0"
I20260806 02:45:50.201646 12030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:50.223820 12030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:50.227600 12030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:50.227859 12030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:50.228621 12030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:50.229669 12030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:50.229718 12030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:50.229759 12030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:50.229774 12030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:50.240842 12030 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.129:34391
I20260806 02:45:50.240922 12331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.129:34391 every 8 connection(s)
I20260806 02:45:50.242004 12030 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
I20260806 02:45:50.251349 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 12030
I20260806 02:45:50.251576 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 11268
Aug 06 02:45:50 dist-test-slave-gmjp krb5kdc[10919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984350, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.189@KRBTEST.COM
Aug 06 02:45:50 dist-test-slave-gmjp krb5kdc[10919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984350, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM
Aug 06 02:45:50 dist-test-slave-gmjp krb5kdc[10919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984350, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.188@KRBTEST.COM
I20260806 02:45:50.268803 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.244258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:48991 (local address 127.1.183.189:40231)
0806 02:45:50.244398 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:50.244402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:50.245370 (+ 968us) server_negotiation.cc:407] Connection header received
0806 02:45:50.246653 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:50.246657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:50.246726 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:50.246804 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:50.248247 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.249082 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.250983 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.251934 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.252054 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.264031 (+ 11977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:50.264038 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:50.264045 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:50.264085 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:50.265942 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:50.266664 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:50.266671 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:50.266675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:50.266767 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:50.267212 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:50.267218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:50.267222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:50.267528 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:50.267626 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:50.268463 (+ 837us) server_negotiation.cc:299] Negotiation successful
0806 02:45:50.268623 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:45:50.269109 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.130:33151
--local_ip_for_outbound_sockets=127.1.183.130
--tserver_master_addrs=127.1.183.190:39091
--webserver_port=46091
--webserver_interface=127.1.183.130
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.130.keytab
--principal=oryx/127.1.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:39639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.183.190:39091,127.1.183.189:40231
--tserver_master_addrs=127.1.183.190:39091,127.1.183.189:40231,127.1.183.188:35427 with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:45:50.269160 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.244233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:60647 (local address 127.1.183.190:39091)
0806 02:45:50.244384 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:50.244390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:50.245372 (+ 982us) server_negotiation.cc:407] Connection header received
0806 02:45:50.246927 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:50.246930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:50.246982 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:50.247070 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:50.248230 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.248772 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.250485 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.250801 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.250882 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.264497 (+ 13615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:50.264502 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:50.264506 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:50.264544 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:50.267053 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:50.267723 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:50.267726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:50.267728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:50.267814 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:50.268102 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:50.268105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:50.268107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:50.268337 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:50.268435 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:50.268875 (+ 440us) server_negotiation.cc:299] Negotiation successful
0806 02:45:50.268999 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 02:45:50.270300 12342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.245799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:35427 (local address 127.1.183.129:45747)
0806 02:45:50.246076 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:50.246091 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:50.246449 (+ 358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:50.247406 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:50.247409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:50.247589 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:50.248069 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.248078 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.249405 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.249422 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.250081 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.250090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.250656 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.250659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.250804 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.251567 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:50.251594 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:50.264932 (+ 13338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:50.267192 (+ 2260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:50.267195 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:50.267197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:50.267478 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:50.267858 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:50.267864 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:50.267868 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:50.267969 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:50.268479 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:50.268482 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:50.268632 (+ 150us) client_negotiation.cc:769] Sending connection context
0806 02:45:50.268765 (+ 133us) client_negotiation.cc:240] Negotiation successful
0806 02:45:50.269149 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":66,"spinlock_wait_cycles":13568,"thread_start_us":80,"threads_started":1}
I20260806 02:45:50.270351 12341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.244678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:40231 (local address 127.1.183.129:48991)
0806 02:45:50.245211 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:50.245264 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:50.246449 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:50.246971 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:50.246977 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:50.247470 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:50.248071 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.248078 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.249197 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.249200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.250360 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.250371 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.260643 (+ 10272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.260647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.260767 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.261361 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:50.261378 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:50.263762 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:50.266093 (+ 2331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:50.266103 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:50.266126 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:50.266538 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:50.266879 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:50.266885 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:50.266890 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:50.267096 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:50.267710 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:50.267721 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:50.268120 (+ 399us) client_negotiation.cc:769] Sending connection context
0806 02:45:50.269735 (+ 1615us) client_negotiation.cc:240] Negotiation successful
0806 02:45:50.269921 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":258,"spinlock_wait_cycles":35072,"thread_start_us":185,"threads_started":1}
I20260806 02:45:50.270700 12340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.244678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.1.183.129:60647)
0806 02:45:50.245209 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:50.245264 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:50.246439 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:50.247095 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:50.247098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:50.247482 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:50.248053 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.248067 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.248917 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.248921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.250360 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.250371 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.250913 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.250918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.251042 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.262095 (+ 11053us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:50.262108 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:50.264275 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:50.267342 (+ 3067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:50.267348 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:50.267351 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:50.267604 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:50.267923 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:50.267927 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:50.267929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:50.267999 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:50.268488 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:50.268490 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:50.268658 (+ 168us) client_negotiation.cc:769] Sending connection context
0806 02:45:50.269300 (+ 642us) client_negotiation.cc:240] Negotiation successful
0806 02:45:50.269450 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":176,"spinlock_wait_cycles":4736,"thread_start_us":159,"threads_started":1}
I20260806 02:45:50.271095 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.245927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:45747 (local address 127.1.183.188:35427)
0806 02:45:50.246221 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:50.246226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:50.246243 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:45:50.247206 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:50.247212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:50.247271 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:50.247365 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:50.248231 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.249298 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.250206 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.250506 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.250607 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.265136 (+ 14529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:50.265141 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:50.265145 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:50.265187 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:50.267053 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:50.267602 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:50.267606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:50.267609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:50.267748 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:50.268077 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:50.268083 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:50.268087 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:50.268338 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:50.268429 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:50.270778 (+ 2349us) server_negotiation.cc:299] Negotiation successful
0806 02:45:50.270893 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":96,"threads_started":1}
I20260806 02:45:50.272074 12332 heartbeater.cc:347] Connected to a master server at 127.1.183.190:39091
I20260806 02:45:50.272158 12333 heartbeater.cc:347] Connected to a master server at 127.1.183.188:35427
I20260806 02:45:50.272351 12332 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:50.272408 12335 heartbeater.cc:347] Connected to a master server at 127.1.183.189:40231
I20260806 02:45:50.272459 12335 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:50.272351 12333 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:50.273020 12333 heartbeater.cc:511] Master 127.1.183.188:35427 requested a full tablet report, sending...
I20260806 02:45:50.273020 12335 heartbeater.cc:511] Master 127.1.183.189:40231 requested a full tablet report, sending...
I20260806 02:45:50.273695 12332 heartbeater.cc:511] Master 127.1.183.190:39091 requested a full tablet report, sending...
I20260806 02:45:50.274824 11828 ts_manager.cc:194] Registered new tserver with Master: fff12eae637c41ebabb8689441a6dbe5 (127.1.183.129:34391)
I20260806 02:45:50.274864 12090 ts_manager.cc:194] Registered new tserver with Master: fff12eae637c41ebabb8689441a6dbe5 (127.1.183.129:34391)
I20260806 02:45:50.274883 10983 ts_manager.cc:194] Re-registered known tserver with Master: fff12eae637c41ebabb8689441a6dbe5 (127.1.183.129:34391)
I20260806 02:45:50.275816 10983 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.183.129@KRBTEST.COM'} at 127.1.183.129:60647
I20260806 02:45:50.339732 11627 heartbeater.cc:503] Master 127.1.183.190:39091 was elected leader, sending a full tablet report...
W20260806 02:45:50.393216 12344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:50.393517 12344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:50.393564 12344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:50.397907 12344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:50.397981 12344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:50.398042 12344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.130
I20260806 02:45:50.403657 12344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:39639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.130.keytab
--principal=oryx/127.1.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.130:33151
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.183.130
--webserver_port=46091
--tserver_master_addrs=127.1.183.190:39091,127.1.183.189:40231,127.1.183.188:35427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.130
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:50.404965 12344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:50.406132 12344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:50.416795 12352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:50.416771 12351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:50.416829 12344 server_base.cc:1034] running on GCE node
W20260806 02:45:50.416771 12356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:50.417582 12344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:50.418303 12344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:50.419503 12344 hybrid_clock.cc:648] HybridClock initialized: now 1785984350419487 us; error 30 us; skew 500 ppm
Aug 06 02:45:50 dist-test-slave-gmjp krb5kdc[10919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984350, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:50.422621 12344 init.cc:377] Logged in from keytab as oryx/127.1.183.130@KRBTEST.COM (short username oryx)
I20260806 02:45:50.423687 12344 webserver.cc:460] Webserver started at http://127.1.183.130:46091/ using document root <none> and password file <none>
I20260806 02:45:50.424391 12344 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:50.424538 12344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:50.429016 12344 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:45:50.431355 12365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:50.432828 12344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:50.432940 12344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-1/wal
uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2"
format_stamp: "Formatted at 2026-08-06 02:45:49 on dist-test-slave-gmjp"
server_key: "db8ed42e1ceeaa1a4f890e40c3bd5b62"
server_key_iv: "2a0ba49b2ab6c82de1071f668c227ee6"
server_key_version: "encryptionkey@0"
I20260806 02:45:50.433511 12344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:50.446246 12344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:50.450059 12344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:50.450235 12344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:50.450873 12344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:50.451763 12344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:50.451812 12344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:50.451855 12344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:50.451871 12344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:50.464056 12344 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.130:33151
I20260806 02:45:50.464077 12551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.130:33151 every 8 connection(s)
I20260806 02:45:50.465206 12344 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
I20260806 02:45:50.468079 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 12344
I20260806 02:45:50.468411 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 11444
Aug 06 02:45:50 dist-test-slave-gmjp krb5kdc[10919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984350, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.188@KRBTEST.COM
I20260806 02:45:50.479815 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.131:39727
--local_ip_for_outbound_sockets=127.1.183.131
--tserver_master_addrs=127.1.183.190:39091
--webserver_port=36435
--webserver_interface=127.1.183.131
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.131.keytab
--principal=oryx/127.1.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:39639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.183.190:39091,127.1.183.189:40231
--tserver_master_addrs=127.1.183.190:39091,127.1.183.189:40231,127.1.183.188:35427 with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:45:50 dist-test-slave-gmjp krb5kdc[10919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984350, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.189@KRBTEST.COM
Aug 06 02:45:50 dist-test-slave-gmjp krb5kdc[10919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984350, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM
I20260806 02:45:50.482810 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.467392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:50917 (local address 127.1.183.188:35427)
0806 02:45:50.467590 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:50.467596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:50.468576 (+ 980us) server_negotiation.cc:407] Connection header received
0806 02:45:50.469925 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:50.469931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:50.470001 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:50.470056 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:50.471731 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.472537 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.474277 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.474651 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.474740 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.478013 (+ 3273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:50.478017 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:50.478020 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:50.478052 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:50.479927 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:50.480768 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:50.480772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:50.480775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:50.480866 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:50.481341 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:50.481345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:50.481349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:50.481574 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:50.481666 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:50.482417 (+ 751us) server_negotiation.cc:299] Negotiation successful
0806 02:45:50.482592 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 02:45:50.484750 12558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.467798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:35427 (local address 127.1.183.130:50917)
0806 02:45:50.468416 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:50.468469 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:50.469685 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:50.470448 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:50.470458 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:50.470956 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:50.471545 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.471565 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.472717 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.472721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.474125 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.474141 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.474964 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.474967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.475130 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.475788 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:50.475811 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:50.477808 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:50.480065 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:50.480073 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:50.480092 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:50.480644 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:50.481014 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:50.481018 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:50.481020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:50.481208 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:50.481737 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:50.481744 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:50.482103 (+ 359us) client_negotiation.cc:769] Sending connection context
0806 02:45:50.483019 (+ 916us) client_negotiation.cc:240] Negotiation successful
0806 02:45:50.483457 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":211,"threads_started":1}
I20260806 02:45:50.485257 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.467416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:33479 (local address 127.1.183.189:40231)
0806 02:45:50.467589 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:50.467594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:50.473200 (+ 5606us) server_negotiation.cc:407] Connection header received
0806 02:45:50.473394 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:50.473401 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:50.473466 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:50.473527 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:50.474323 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.475183 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.476627 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.477093 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.477192 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.480745 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:50.480752 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:50.480758 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:50.480801 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:50.482749 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:50.483620 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:50.483627 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:50.483631 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:50.483724 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:50.484066 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:50.484072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:50.484077 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:50.484360 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:50.484455 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:50.484859 (+ 404us) server_negotiation.cc:299] Negotiation successful
0806 02:45:50.485012 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 02:45:50.486054 12555 heartbeater.cc:347] Connected to a master server at 127.1.183.188:35427
I20260806 02:45:50.486333 12559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.467820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:40231 (local address 127.1.183.130:33479)
0806 02:45:50.473093 (+ 5273us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:50.473111 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:50.473278 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:50.473695 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:50.473700 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:50.473973 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:50.474155 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.474168 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.475357 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.475362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.476485 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.476500 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.477452 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.477459 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.477590 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.478646 (+ 1056us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:50.478667 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:50.480566 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:50.483167 (+ 2601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:50.483170 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:50.483175 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:50.483497 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:50.483853 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:50.483858 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:50.483863 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:50.483958 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:50.484521 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:50.484526 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:50.484634 (+ 108us) client_negotiation.cc:769] Sending connection context
0806 02:45:50.485718 (+ 1084us) client_negotiation.cc:240] Negotiation successful
0806 02:45:50.485914 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5182,"mutex_wait_us":23,"spinlock_wait_cycles":60544,"thread_start_us":198,"threads_started":1}
I20260806 02:45:50.486341 12555 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:50.486702 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.471451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:55333 (local address 127.1.183.190:39091)
0806 02:45:50.471608 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:50.471615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:50.475676 (+ 4061us) server_negotiation.cc:407] Connection header received
0806 02:45:50.475912 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:50.475916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:50.475976 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:50.476036 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:50.476752 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.477581 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.478853 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.479278 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.479368 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.482051 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:50.482055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:50.482057 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:50.482085 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:50.483887 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:50.484430 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:50.484434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:50.484436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:50.484540 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:50.484881 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:50.484885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:50.484887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:50.485107 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:50.485202 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:50.486443 (+ 1241us) server_negotiation.cc:299] Negotiation successful
0806 02:45:50.486558 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:45:50.487207 12561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.469009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.1.183.130:55333)
0806 02:45:50.475575 (+ 6566us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:50.475591 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:50.475755 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:50.476233 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:50.476236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:50.476426 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:50.476597 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.476609 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.477754 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.477757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.478701 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.478716 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.479592 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.479596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.479713 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.480311 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:50.480326 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:50.481893 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:50.484031 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:50.484034 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:50.484036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:50.484299 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:50.484653 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:50.484656 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:50.484658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:50.484764 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:50.486130 (+ 1366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:50.486136 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:50.486230 (+ 94us) client_negotiation.cc:769] Sending connection context
0806 02:45:50.486900 (+ 670us) client_negotiation.cc:240] Negotiation successful
0806 02:45:50.487043 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6484,"mutex_wait_us":1,"thread_start_us":1678,"threads_started":1}
I20260806 02:45:50.487295 12555 heartbeater.cc:511] Master 127.1.183.188:35427 requested a full tablet report, sending...
I20260806 02:45:50.487794 12552 heartbeater.cc:347] Connected to a master server at 127.1.183.190:39091
I20260806 02:45:50.487890 12552 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:50.488001 12556 heartbeater.cc:347] Connected to a master server at 127.1.183.189:40231
I20260806 02:45:50.488107 12552 heartbeater.cc:511] Master 127.1.183.190:39091 requested a full tablet report, sending...
I20260806 02:45:50.488111 12556 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:50.488313 12556 heartbeater.cc:511] Master 127.1.183.189:40231 requested a full tablet report, sending...
I20260806 02:45:50.489084 12090 ts_manager.cc:194] Registered new tserver with Master: 5e8dc07e3a1f4de49b0e6a5bec6878f2 (127.1.183.130:33151)
I20260806 02:45:50.489174 11828 ts_manager.cc:194] Registered new tserver with Master: 5e8dc07e3a1f4de49b0e6a5bec6878f2 (127.1.183.130:33151)
I20260806 02:45:50.489512 10983 ts_manager.cc:194] Re-registered known tserver with Master: 5e8dc07e3a1f4de49b0e6a5bec6878f2 (127.1.183.130:33151)
I20260806 02:45:50.490154 10983 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.183.130@KRBTEST.COM'} at 127.1.183.130:55333
I20260806 02:45:50.608328 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.604492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37750 (local address 127.1.183.188:35427)
0806 02:45:50.604634 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:50.604638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:50.604864 (+ 226us) server_negotiation.cc:407] Connection header received
0806 02:45:50.605044 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:50.605047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:50.605115 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:50.605213 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:50.605511 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.606120 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.607218 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.607749 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.607875 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.608041 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:50.608083 (+ 42us) server_negotiation.cc:299] Negotiation successful
0806 02:45:50.608149 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:45:50.608333 12566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.604428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:35427 (local address 127.0.0.1:37750)
0806 02:45:50.604760 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:50.604773 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:50.604922 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:50.605239 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:50.605242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:50.605261 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:50.605407 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.605417 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.606231 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.606235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.607088 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.607098 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.607883 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.607887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.607978 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.608002 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 02:45:50.608071 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 02:45:50.608150 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":76,"threads_started":1}
I20260806 02:45:50.609263 12093 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260806 02:45:50.609892 12145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5975de0199eb4bc1a2ac5d5607e1928c. Latest consensus state: current_term: 1 leader_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: false } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: NON_VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: true } } }
I20260806 02:45:50.610046 12145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [sys.catalog]: This master's current role is: LEARNER
W20260806 02:45:50.625277 12563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:50.625562 12563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:50.625608 12563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:50.630065 12563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:50.630132 12563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:50.630193 12563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.131
I20260806 02:45:50.635877 12563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:39639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.131.keytab
--principal=oryx/127.1.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.131:39727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.183.131
--webserver_port=36435
--tserver_master_addrs=127.1.183.190:39091,127.1.183.189:40231,127.1.183.188:35427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.131
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:50.637244 12563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:50.638485 12563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:50.649201 12571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:50.649211 12575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:50.649214 12572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:50.649644 12563 server_base.cc:1034] running on GCE node
I20260806 02:45:50.650094 12563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:50.650795 12563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:50.651978 12563 hybrid_clock.cc:648] HybridClock initialized: now 1785984350651952 us; error 30 us; skew 500 ppm
Aug 06 02:45:50 dist-test-slave-gmjp krb5kdc[10919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984350, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:50.655031 12563 init.cc:377] Logged in from keytab as oryx/127.1.183.131@KRBTEST.COM (short username oryx)
I20260806 02:45:50.656003 12563 webserver.cc:460] Webserver started at http://127.1.183.131:36435/ using document root <none> and password file <none>
I20260806 02:45:50.656680 12563 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:50.656793 12563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:50.660938 12563 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:45:50.663232 12586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:50.664657 12563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:50.664811 12563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-2/wal
uuid: "e2cbd8619d9048b5b56e4d9ea417288e"
format_stamp: "Formatted at 2026-08-06 02:45:49 on dist-test-slave-gmjp"
server_key: "d1a8b79dca2874c0395e30afffec4812"
server_key_iv: "b3611f31a0682499d1b0562c3457a30f"
server_key_version: "encryptionkey@0"
I20260806 02:45:50.665407 12563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:50.688882 12563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:50.692250 12563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:50.692480 12563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:50.693135 12563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:45:50.694070 12563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:50.694145 12563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:50.694221 12563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:50.694264 12563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:50.704855 12563 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.131:39727
I20260806 02:45:50.704909 12754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.131:39727 every 8 connection(s)
I20260806 02:45:50.706147 12563 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
I20260806 02:45:50.709893 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 12563
Aug 06 02:45:50 dist-test-slave-gmjp krb5kdc[10919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984350, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.189@KRBTEST.COM
Aug 06 02:45:50 dist-test-slave-gmjp krb5kdc[10919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984350, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM
Aug 06 02:45:50 dist-test-slave-gmjp krb5kdc[10919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984350, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.188@KRBTEST.COM
I20260806 02:45:50.726981 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.708326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:34335 (local address 127.1.183.188:35427)
0806 02:45:50.708457 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:50.708462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:50.709359 (+ 897us) server_negotiation.cc:407] Connection header received
0806 02:45:50.710618 (+ 1259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:50.710622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:50.710685 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:50.710744 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:50.717428 (+ 6684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.718213 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.719385 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.719711 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.719815 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.722361 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:50.722368 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:50.722373 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:50.722417 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:50.724215 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:50.724810 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:50.724815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:50.724818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:50.724906 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:50.725373 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:50.725399 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:50.725401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:50.725667 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:50.725773 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:50.726646 (+ 873us) server_negotiation.cc:299] Negotiation successful
0806 02:45:50.726793 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 02:45:50.727208 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.708089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:56159 (local address 127.1.183.189:40231)
0806 02:45:50.708283 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:50.708288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:50.709408 (+ 1120us) server_negotiation.cc:407] Connection header received
0806 02:45:50.710693 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:50.710697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:50.710759 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:50.710851 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:50.712896 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.713731 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.715936 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.716301 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.716380 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.720935 (+ 4555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:50.720939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:50.720942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:50.720972 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:50.723352 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:50.724924 (+ 1572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:50.724929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:50.724933 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:50.725028 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:50.725435 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:50.725439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:50.725441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:50.725709 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:50.725812 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:50.726667 (+ 855us) server_negotiation.cc:299] Negotiation successful
0806 02:45:50.726821 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 02:45:50.727010 12773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.712380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37754 (local address 127.1.183.188:35427)
0806 02:45:50.712905 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:50.712911 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:50.714131 (+ 1220us) server_negotiation.cc:407] Connection header received
0806 02:45:50.714334 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:50.714337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:50.714390 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:50.714461 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:50.715412 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.716268 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.718257 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.720026 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.720123 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.722841 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:50.722845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:50.722847 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:50.722876 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:50.725130 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:50.725669 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:50.725673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:50.725675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:50.725754 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:50.726104 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:50.726107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:50.726109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:50.726298 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:50.726406 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:50.726774 (+ 368us) server_negotiation.cc:299] Negotiation successful
0806 02:45:50.726871 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":90,"threads_started":1}
I20260806 02:45:50.729487 12761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.708671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:40231 (local address 127.1.183.131:56159)
0806 02:45:50.709308 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:50.709324 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:50.710367 (+ 1043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:50.711076 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:50.711085 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:50.711741 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:50.712695 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.712718 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.713932 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.713935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.715781 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.715793 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.716601 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.716605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.716921 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.717827 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:50.717848 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:50.720705 (+ 2857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:50.724052 (+ 3347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:50.724062 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:50.724085 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:50.724807 (+ 722us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:50.725205 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:50.725209 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:50.725212 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:50.725311 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:50.725925 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:50.725929 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:50.726341 (+ 412us) client_negotiation.cc:769] Sending connection context
0806 02:45:50.727396 (+ 1055us) client_negotiation.cc:240] Negotiation successful
0806 02:45:50.728250 (+ 854us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":1054,"spinlock_wait_cycles":42624,"thread_start_us":148,"threads_started":1}
I20260806 02:45:50.729991 12763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.708671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:35427 (local address 127.1.183.131:34335)
0806 02:45:50.709199 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:50.709252 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:50.710362 (+ 1110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:50.711076 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:50.711085 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:50.711741 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:50.716622 (+ 4881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.716910 (+ 288us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.718353 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.718356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.719275 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.719287 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.719849 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.719853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.719989 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.720637 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:50.720652 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:50.722187 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:50.724350 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:50.724353 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:50.724357 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:50.724677 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:50.725031 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:50.725034 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:50.725036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:50.725261 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:50.725907 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:50.725916 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:50.726339 (+ 423us) client_negotiation.cc:769] Sending connection context
0806 02:45:50.727186 (+ 847us) client_negotiation.cc:240] Negotiation successful
0806 02:45:50.727614 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":137,"spinlock_wait_cycles":44544,"thread_start_us":157,"threads_started":1}
I20260806 02:45:50.730634 12765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.709746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.1.183.131:41717)
0806 02:45:50.710066 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:50.710082 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:50.710362 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:50.712520 (+ 2158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:50.712524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:50.712711 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:50.712847 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.712855 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.713932 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.713935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.715782 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.715793 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.716601 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.716605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.716919 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.718681 (+ 1762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:50.718698 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:50.724581 (+ 5883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:50.728558 (+ 3977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:50.728561 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:50.728563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:50.728826 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:50.729431 (+ 605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:50.729436 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:50.729439 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:50.729545 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:50.730048 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:50.730053 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:50.730151 (+ 98us) client_negotiation.cc:769] Sending connection context
0806 02:45:50.730304 (+ 153us) client_negotiation.cc:240] Negotiation successful
0806 02:45:50.730472 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":249,"spinlock_wait_cycles":1280,"thread_start_us":90,"threads_started":1}
I20260806 02:45:50.730772 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.711978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:41717 (local address 127.1.183.190:39091)
0806 02:45:50.712265 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:50.712273 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:50.712287 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:45:50.712333 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:50.712336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:50.712383 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:50.712475 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:50.712984 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.713746 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.715926 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.716301 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.716388 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.724741 (+ 8353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:50.724748 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:50.724753 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:50.724784 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:50.726680 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:50.728916 (+ 2236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:50.728921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:50.728925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:50.728996 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:50.729671 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:50.729674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:50.729677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:50.729892 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:50.729969 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:50.730389 (+ 420us) server_negotiation.cc:299] Negotiation successful
0806 02:45:50.730622 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":206}
I20260806 02:45:50.730954 12757 heartbeater.cc:347] Connected to a master server at 127.1.183.189:40231
I20260806 02:45:50.730953 12756 heartbeater.cc:347] Connected to a master server at 127.1.183.188:35427
I20260806 02:45:50.731253 12756 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:50.731253 12757 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:50.731557 12774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.712077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53568 (local address 127.1.183.190:39091)
0806 02:45:50.712751 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:50.712757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:50.713202 (+ 445us) server_negotiation.cc:407] Connection header received
0806 02:45:50.713245 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:50.713248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:50.713314 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:50.713396 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:50.714483 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.715350 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.716518 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.716842 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.716942 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.722426 (+ 5484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:50.722431 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:50.722433 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:50.722465 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:50.727676 (+ 5211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:50.728290 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:50.728296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:50.728299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:50.728390 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:50.728773 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:50.728777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:50.728779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:50.729221 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:50.729378 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:50.729927 (+ 549us) server_negotiation.cc:299] Negotiation successful
0806 02:45:50.731397 (+ 1470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":497,"threads_started":1}
I20260806 02:45:50.731951 12757 heartbeater.cc:511] Master 127.1.183.189:40231 requested a full tablet report, sending...
I20260806 02:45:50.732100 12755 heartbeater.cc:347] Connected to a master server at 127.1.183.190:39091
I20260806 02:45:50.732216 12755 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:50.732416 12755 heartbeater.cc:511] Master 127.1.183.190:39091 requested a full tablet report, sending...
I20260806 02:45:50.731933 12756 heartbeater.cc:511] Master 127.1.183.188:35427 requested a full tablet report, sending...
I20260806 02:45:50.733362 11828 ts_manager.cc:194] Registered new tserver with Master: e2cbd8619d9048b5b56e4d9ea417288e (127.1.183.131:39727)
I20260806 02:45:50.733646 10983 ts_manager.cc:194] Re-registered known tserver with Master: e2cbd8619d9048b5b56e4d9ea417288e (127.1.183.131:39727)
I20260806 02:45:50.733856 12090 ts_manager.cc:194] Registered new tserver with Master: e2cbd8619d9048b5b56e4d9ea417288e (127.1.183.131:39727)
I20260806 02:45:50.734405 10983 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.183.131@KRBTEST.COM'} at 127.1.183.131:41717
I20260806 02:45:50.737737 12777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.713220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51224 (local address 127.1.183.189:40231)
0806 02:45:50.713825 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:50.713833 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:50.717663 (+ 3830us) server_negotiation.cc:407] Connection header received
0806 02:45:50.717709 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:50.717712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:50.717769 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:50.717854 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:50.721567 (+ 3713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.722422 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.723598 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.723997 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.724092 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.729079 (+ 4987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:50.729084 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:50.729086 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:50.729110 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:50.735736 (+ 6626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:50.736351 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:50.736354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:50.736357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:50.736457 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:50.736762 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:50.736765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:50.736766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:50.736958 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:50.737067 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:50.737374 (+ 307us) server_negotiation.cc:299] Negotiation successful
0806 02:45:50.737508 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":74,"threads_started":1}
I20260806 02:45:50.736027 10983 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:45:50.740276 10983 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:45:50.743306 12781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: false } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: NON_VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: true } } }
I20260806 02:45:50.743489 12781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:50.747246 12781 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c: attempting to promote NON_VOTER 938ea1eb1f524f2eacb24a0224ce4694 to VOTER
W20260806 02:45:50.747576 12782 consensus_queue.cc:1159] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" responder_term: 1 status { last_received { term: 1 index: 8 } last_committed_idx: 8 last_received_current_leader { term: 1 index: 8 } }
I20260806 02:45:50.747846 12781 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: false } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: false } }
I20260806 02:45:50.749094 12093 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [term 1 LEARNER]: Refusing update from remote peer 5975de0199eb4bc1a2ac5d5607e1928c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260806 02:45:50.749552 12782 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [LEADER]: Connected to new peer: Peer: permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 02:45:50.750360 11843 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [term 1 FOLLOWER]: Refusing update from remote peer 5975de0199eb4bc1a2ac5d5607e1928c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260806 02:45:50.751399 12781 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 02:45:50.751327 12565 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 938ea1eb1f524f2eacb24a0224ce4694 (127.1.183.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: false } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: false } } }
I20260806 02:45:50.752086 12093 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 938ea1eb1f524f2eacb24a0224ce4694 (127.1.183.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: false } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: false } } }
I20260806 02:45:50.753103 12565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: false } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: false } } }
I20260806 02:45:50.753239 12565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:50.753387 12565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: false } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: false } } }
I20260806 02:45:50.753472 12565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:50.753589 11843 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 938ea1eb1f524f2eacb24a0224ce4694 (127.1.183.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: false } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: false } } }
I20260806 02:45:50.754410 12145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: false } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: false } } }
I20260806 02:45:50.754539 12145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:45:50.756467 12791 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: false } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: false } } }
I20260806 02:45:50.756628 12791 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:45:50.756142 12565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: false } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: false } } }
I20260806 02:45:50.757390 12565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:50.764048 12566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.757698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:34391 (local address 127.0.0.1:35854)
0806 02:45:50.757911 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:50.757932 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:50.758028 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:50.759018 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:50.759022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:50.759042 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:50.759158 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.759166 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.760967 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.760971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.762346 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.762358 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.763593 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.763598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.763698 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.763743 (+ 45us) client_negotiation.cc:769] Sending connection context
0806 02:45:50.763802 (+ 59us) client_negotiation.cc:240] Negotiation successful
0806 02:45:50.763899 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260806 02:45:50.764671 12794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.758298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:39727 (local address 127.0.0.1:54994)
0806 02:45:50.759468 (+ 1170us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:50.759477 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:50.759574 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:50.760040 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:50.760043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:50.760061 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:50.760183 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.760191 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.761528 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.761532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.762384 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.762394 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.764274 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.764277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.764366 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.764384 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 02:45:50.764469 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 02:45:50.764540 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1130,"thread_start_us":97,"threads_started":1}
I20260806 02:45:50.764755 12798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.758745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54994 (local address 127.1.183.131:39727)
0806 02:45:50.759078 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:50.759090 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:50.759571 (+ 481us) server_negotiation.cc:407] Connection header received
0806 02:45:50.759691 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:50.759697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:50.759898 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:50.760056 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:50.760282 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.761424 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.762516 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.763393 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.763493 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.763646 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:50.764508 (+ 862us) server_negotiation.cc:299] Negotiation successful
0806 02:45:50.764614 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":138,"threads_started":1}
I20260806 02:45:50.764962 12792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.757870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35854 (local address 127.1.183.129:34391)
0806 02:45:50.758404 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:50.758422 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:50.758450 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:45:50.758566 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:50.758573 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:50.758835 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:50.759340 (+ 505us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:50.759495 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.760821 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.762523 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.763441 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.764423 (+ 982us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.764597 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:50.764655 (+ 58us) server_negotiation.cc:299] Negotiation successful
0806 02:45:50.764789 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":100,"threads_started":1}
I20260806 02:45:50.766294 12795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.757956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:33151 (local address 127.0.0.1:36926)
0806 02:45:50.759249 (+ 1293us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:50.759261 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:50.759338 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:50.760607 (+ 1269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:50.760610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:50.760628 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:50.760706 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.760712 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.763245 (+ 2533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.763249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.764117 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.764127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.765938 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.765942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.766029 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.766054 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 02:45:50.766104 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 02:45:50.766169 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1247,"thread_start_us":145,"threads_started":1}
I20260806 02:45:50.767024 12796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.758475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36926 (local address 127.1.183.130:33151)
0806 02:45:50.759827 (+ 1352us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:50.759838 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:50.759859 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:45:50.759933 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:50.759938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:50.760450 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:50.760840 (+ 390us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:50.760970 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.763111 (+ 2141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.764974 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.765797 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.766582 (+ 785us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.766731 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:50.766783 (+ 52us) server_negotiation.cc:299] Negotiation successful
0806 02:45:50.766880 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1252,"thread_start_us":129,"threads_started":1}
I20260806 02:45:50.768101 12248 tablet_service.cc:1467] Processing CreateTablet for tablet 34a2905691c8446cb9774cc315d77934 (DEFAULT_TABLE table=test-table [id=87e5fe873c154ab29a44c33ff1c7cf72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:50.768090 12657 tablet_service.cc:1467] Processing CreateTablet for tablet 34a2905691c8446cb9774cc315d77934 (DEFAULT_TABLE table=test-table [id=87e5fe873c154ab29a44c33ff1c7cf72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:50.769330 12248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34a2905691c8446cb9774cc315d77934. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:50.769454 12657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34a2905691c8446cb9774cc315d77934. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:50.769763 12439 tablet_service.cc:1467] Processing CreateTablet for tablet 34a2905691c8446cb9774cc315d77934 (DEFAULT_TABLE table=test-table [id=87e5fe873c154ab29a44c33ff1c7cf72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:50.770778 12439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34a2905691c8446cb9774cc315d77934. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:50.779600 12801 tablet_bootstrap.cc:492] T 34a2905691c8446cb9774cc315d77934 P fff12eae637c41ebabb8689441a6dbe5: Bootstrap starting.
I20260806 02:45:50.779600 12800 tablet_bootstrap.cc:492] T 34a2905691c8446cb9774cc315d77934 P e2cbd8619d9048b5b56e4d9ea417288e: Bootstrap starting.
I20260806 02:45:50.781131 12799 tablet_bootstrap.cc:492] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2: Bootstrap starting.
I20260806 02:45:50.782220 12801 tablet_bootstrap.cc:654] T 34a2905691c8446cb9774cc315d77934 P fff12eae637c41ebabb8689441a6dbe5: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:50.782227 12800 tablet_bootstrap.cc:654] T 34a2905691c8446cb9774cc315d77934 P e2cbd8619d9048b5b56e4d9ea417288e: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:50.783062 12801 log.cc:824] T 34a2905691c8446cb9774cc315d77934 P fff12eae637c41ebabb8689441a6dbe5: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:50.783119 12800 log.cc:824] T 34a2905691c8446cb9774cc315d77934 P e2cbd8619d9048b5b56e4d9ea417288e: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:50.783324 12799 tablet_bootstrap.cc:654] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:50.784286 12799 log.cc:824] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:50.785420 12800 tablet_bootstrap.cc:492] T 34a2905691c8446cb9774cc315d77934 P e2cbd8619d9048b5b56e4d9ea417288e: No bootstrap required, opened a new log
I20260806 02:45:50.785439 12801 tablet_bootstrap.cc:492] T 34a2905691c8446cb9774cc315d77934 P fff12eae637c41ebabb8689441a6dbe5: No bootstrap required, opened a new log
I20260806 02:45:50.785629 12800 ts_tablet_manager.cc:1397] T 34a2905691c8446cb9774cc315d77934 P e2cbd8619d9048b5b56e4d9ea417288e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:45:50.785651 12801 ts_tablet_manager.cc:1397] T 34a2905691c8446cb9774cc315d77934 P fff12eae637c41ebabb8689441a6dbe5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:45:50.786389 12799 tablet_bootstrap.cc:492] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2: No bootstrap required, opened a new log
I20260806 02:45:50.786581 12799 ts_tablet_manager.cc:1397] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260806 02:45:50.789240 12801 raft_consensus.cc:348] T 34a2905691c8446cb9774cc315d77934 P fff12eae637c41ebabb8689441a6dbe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff12eae637c41ebabb8689441a6dbe5" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 34391 } } peers { permanent_uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33151 } } peers { permanent_uuid: "e2cbd8619d9048b5b56e4d9ea417288e" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 39727 } }
I20260806 02:45:50.789543 12801 raft_consensus.cc:374] T 34a2905691c8446cb9774cc315d77934 P fff12eae637c41ebabb8689441a6dbe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:50.789618 12801 raft_consensus.cc:729] T 34a2905691c8446cb9774cc315d77934 P fff12eae637c41ebabb8689441a6dbe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fff12eae637c41ebabb8689441a6dbe5, State: Initialized, Role: FOLLOWER
I20260806 02:45:50.790170 12801 consensus_queue.cc:260] T 34a2905691c8446cb9774cc315d77934 P fff12eae637c41ebabb8689441a6dbe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff12eae637c41ebabb8689441a6dbe5" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 34391 } } peers { permanent_uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33151 } } peers { permanent_uuid: "e2cbd8619d9048b5b56e4d9ea417288e" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 39727 } }
I20260806 02:45:50.790570 12800 raft_consensus.cc:348] T 34a2905691c8446cb9774cc315d77934 P e2cbd8619d9048b5b56e4d9ea417288e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff12eae637c41ebabb8689441a6dbe5" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 34391 } } peers { permanent_uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33151 } } peers { permanent_uuid: "e2cbd8619d9048b5b56e4d9ea417288e" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 39727 } }
I20260806 02:45:50.790876 12800 raft_consensus.cc:374] T 34a2905691c8446cb9774cc315d77934 P e2cbd8619d9048b5b56e4d9ea417288e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:50.790944 12800 raft_consensus.cc:729] T 34a2905691c8446cb9774cc315d77934 P e2cbd8619d9048b5b56e4d9ea417288e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2cbd8619d9048b5b56e4d9ea417288e, State: Initialized, Role: FOLLOWER
I20260806 02:45:50.791409 12801 ts_tablet_manager.cc:1428] T 34a2905691c8446cb9774cc315d77934 P fff12eae637c41ebabb8689441a6dbe5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:45:50.791407 12800 consensus_queue.cc:260] T 34a2905691c8446cb9774cc315d77934 P e2cbd8619d9048b5b56e4d9ea417288e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff12eae637c41ebabb8689441a6dbe5" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 34391 } } peers { permanent_uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33151 } } peers { permanent_uuid: "e2cbd8619d9048b5b56e4d9ea417288e" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 39727 } }
I20260806 02:45:50.791846 12332 heartbeater.cc:503] Master 127.1.183.190:39091 was elected leader, sending a full tablet report...
I20260806 02:45:50.792493 12800 ts_tablet_manager.cc:1428] T 34a2905691c8446cb9774cc315d77934 P e2cbd8619d9048b5b56e4d9ea417288e: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260806 02:45:50.792878 12755 heartbeater.cc:503] Master 127.1.183.190:39091 was elected leader, sending a full tablet report...
I20260806 02:45:50.791646 12799 raft_consensus.cc:348] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff12eae637c41ebabb8689441a6dbe5" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 34391 } } peers { permanent_uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33151 } } peers { permanent_uuid: "e2cbd8619d9048b5b56e4d9ea417288e" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 39727 } }
I20260806 02:45:50.793727 12799 raft_consensus.cc:374] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:50.793802 12799 raft_consensus.cc:729] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8dc07e3a1f4de49b0e6a5bec6878f2, State: Initialized, Role: FOLLOWER
I20260806 02:45:50.794284 12799 consensus_queue.cc:260] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff12eae637c41ebabb8689441a6dbe5" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 34391 } } peers { permanent_uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33151 } } peers { permanent_uuid: "e2cbd8619d9048b5b56e4d9ea417288e" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 39727 } }
I20260806 02:45:50.795553 12799 ts_tablet_manager.cc:1428] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2: Time spent starting tablet: real 0.009s user 0.000s sys 0.008s
I20260806 02:45:50.796729 12552 heartbeater.cc:503] Master 127.1.183.190:39091 was elected leader, sending a full tablet report...
I20260806 02:45:50.943181 12810 raft_consensus.cc:482] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:45:50.943444 12810 raft_consensus.cc:504] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff12eae637c41ebabb8689441a6dbe5" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 34391 } } peers { permanent_uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33151 } } peers { permanent_uuid: "e2cbd8619d9048b5b56e4d9ea417288e" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 39727 } }
I20260806 02:45:50.944933 12810 leader_election.cc:288] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fff12eae637c41ebabb8689441a6dbe5 (127.1.183.129:34391), e2cbd8619d9048b5b56e4d9ea417288e (127.1.183.131:39727)
I20260806 02:45:50.950031 12559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.945209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:39727 (local address 127.1.183.130:40987)
0806 02:45:50.945496 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:50.945508 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:50.945568 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:50.945848 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:50.945851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:50.945863 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:50.945981 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.945987 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.947020 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.947024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.948649 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.948663 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.949589 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.949592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.949684 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.949717 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 02:45:50.949782 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 02:45:50.949856 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":338}
I20260806 02:45:50.950094 12798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.945432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:40987 (local address 127.1.183.131:39727)
0806 02:45:50.945555 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:50.945561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:50.945574 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:45:50.945655 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:50.945658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:50.945711 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:50.945784 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:50.946085 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.946887 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.948808 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.949438 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.949587 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.949646 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:50.949862 (+ 216us) server_negotiation.cc:299] Negotiation successful
0806 02:45:50.949958 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:45:50.950196 12561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.945214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:34391 (local address 127.1.183.130:54775)
0806 02:45:50.945365 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:50.945382 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:50.945489 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:50.945780 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:50.945784 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:50.945804 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:50.945969 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.945978 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.947020 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.947024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.948649 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.948663 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.949716 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.949720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.949891 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.949916 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 02:45:50.949998 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 02:45:50.950067 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":156}
I20260806 02:45:50.950281 12792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.945338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:54775 (local address 127.1.183.129:34391)
0806 02:45:50.945489 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:50.945495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:50.945513 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:45:50.945598 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:50.945601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:50.945655 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:50.945739 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:50.946090 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.946890 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.948808 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.949464 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.949587 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.949646 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:50.950046 (+ 400us) server_negotiation.cc:299] Negotiation successful
0806 02:45:50.950142 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:45:50.950737 12805 raft_consensus.cc:482] T 34a2905691c8446cb9774cc315d77934 P fff12eae637c41ebabb8689441a6dbe5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:45:50.950955 12805 raft_consensus.cc:504] T 34a2905691c8446cb9774cc315d77934 P fff12eae637c41ebabb8689441a6dbe5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff12eae637c41ebabb8689441a6dbe5" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 34391 } } peers { permanent_uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33151 } } peers { permanent_uuid: "e2cbd8619d9048b5b56e4d9ea417288e" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 39727 } }
I20260806 02:45:50.951169 12281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34a2905691c8446cb9774cc315d77934" candidate_uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fff12eae637c41ebabb8689441a6dbe5" is_pre_election: true
I20260806 02:45:50.951428 12281 raft_consensus.cc:2454] T 34a2905691c8446cb9774cc315d77934 P fff12eae637c41ebabb8689441a6dbe5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8dc07e3a1f4de49b0e6a5bec6878f2 in term 0.
I20260806 02:45:50.951555 12688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34a2905691c8446cb9774cc315d77934" candidate_uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2cbd8619d9048b5b56e4d9ea417288e" is_pre_election: true
I20260806 02:45:50.951833 12688 raft_consensus.cc:2454] T 34a2905691c8446cb9774cc315d77934 P e2cbd8619d9048b5b56e4d9ea417288e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8dc07e3a1f4de49b0e6a5bec6878f2 in term 0.
I20260806 02:45:50.952092 12370 leader_election.cc:302] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8dc07e3a1f4de49b0e6a5bec6878f2, fff12eae637c41ebabb8689441a6dbe5; no voters:
I20260806 02:45:50.952348 12805 leader_election.cc:288] T 34a2905691c8446cb9774cc315d77934 P fff12eae637c41ebabb8689441a6dbe5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e8dc07e3a1f4de49b0e6a5bec6878f2 (127.1.183.130:33151), e2cbd8619d9048b5b56e4d9ea417288e (127.1.183.131:39727)
I20260806 02:45:50.954048 12810 raft_consensus.cc:2781] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:45:50.954150 12810 raft_consensus.cc:482] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:45:50.954193 12810 raft_consensus.cc:3037] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:50.955435 12810 raft_consensus.cc:504] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff12eae637c41ebabb8689441a6dbe5" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 34391 } } peers { permanent_uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33151 } } peers { permanent_uuid: "e2cbd8619d9048b5b56e4d9ea417288e" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 39727 } }
I20260806 02:45:50.955859 12810 leader_election.cc:288] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [CANDIDATE]: Term 1 election: Requested vote from peers fff12eae637c41ebabb8689441a6dbe5 (127.1.183.129:34391), e2cbd8619d9048b5b56e4d9ea417288e (127.1.183.131:39727)
I20260806 02:45:50.956239 12281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34a2905691c8446cb9774cc315d77934" candidate_uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fff12eae637c41ebabb8689441a6dbe5"
I20260806 02:45:50.956239 12688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34a2905691c8446cb9774cc315d77934" candidate_uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2cbd8619d9048b5b56e4d9ea417288e"
I20260806 02:45:50.956372 12281 raft_consensus.cc:3037] T 34a2905691c8446cb9774cc315d77934 P fff12eae637c41ebabb8689441a6dbe5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:50.956387 12688 raft_consensus.cc:3037] T 34a2905691c8446cb9774cc315d77934 P e2cbd8619d9048b5b56e4d9ea417288e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:50.957137 12798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.952706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:60883 (local address 127.1.183.131:39727)
0806 02:45:50.952819 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:50.952823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:50.953319 (+ 496us) server_negotiation.cc:407] Connection header received
0806 02:45:50.953412 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:50.953417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:50.953477 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:50.953605 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:50.953816 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.954509 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.955731 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.956366 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.956479 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.956540 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:50.956916 (+ 376us) server_negotiation.cc:299] Negotiation successful
0806 02:45:50.957010 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 02:45:50.957528 12812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.952580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:39727 (local address 127.1.183.129:60883)
0806 02:45:50.953213 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:50.953225 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:50.953330 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:50.953611 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:50.953614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:50.953631 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:50.953735 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.953739 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.954660 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.954663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.955519 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.955565 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.956602 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.956606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.956746 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.956791 (+ 45us) client_negotiation.cc:769] Sending connection context
0806 02:45:50.957265 (+ 474us) client_negotiation.cc:240] Negotiation successful
0806 02:45:50.957349 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":16,"thread_start_us":130,"threads_started":1}
I20260806 02:45:50.957754 12688 raft_consensus.cc:2454] T 34a2905691c8446cb9774cc315d77934 P e2cbd8619d9048b5b56e4d9ea417288e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8dc07e3a1f4de49b0e6a5bec6878f2 in term 1.
I20260806 02:45:50.957754 12281 raft_consensus.cc:2454] T 34a2905691c8446cb9774cc315d77934 P fff12eae637c41ebabb8689441a6dbe5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8dc07e3a1f4de49b0e6a5bec6878f2 in term 1.
W20260806 02:45:50.958133 12759 tablet.cc:2369] T 34a2905691c8446cb9774cc315d77934 P e2cbd8619d9048b5b56e4d9ea417288e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:45:50.958252 12367 leader_election.cc:302] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8dc07e3a1f4de49b0e6a5bec6878f2, e2cbd8619d9048b5b56e4d9ea417288e; no voters:
I20260806 02:45:50.958691 12688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34a2905691c8446cb9774cc315d77934" candidate_uuid: "fff12eae637c41ebabb8689441a6dbe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2cbd8619d9048b5b56e4d9ea417288e" is_pre_election: true
I20260806 02:45:50.958935 12688 raft_consensus.cc:2379] T 34a2905691c8446cb9774cc315d77934 P e2cbd8619d9048b5b56e4d9ea417288e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fff12eae637c41ebabb8689441a6dbe5 in current term 1: Already voted for candidate 5e8dc07e3a1f4de49b0e6a5bec6878f2 in this term.
I20260806 02:45:50.959223 12810 raft_consensus.cc:2781] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:50.959470 12810 raft_consensus.cc:686] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [term 1 LEADER]: Becoming Leader. State: Replica: 5e8dc07e3a1f4de49b0e6a5bec6878f2, State: Running, Role: LEADER
I20260806 02:45:50.959848 12810 consensus_queue.cc:237] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff12eae637c41ebabb8689441a6dbe5" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 34391 } } peers { permanent_uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33151 } } peers { permanent_uuid: "e2cbd8619d9048b5b56e4d9ea417288e" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 39727 } }
I20260806 02:45:50.960399 12811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.952584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:33151 (local address 127.1.183.129:47597)
0806 02:45:50.956125 (+ 3541us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:50.956141 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:50.956328 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:50.956689 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:50.956693 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:50.956710 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:50.956895 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.956902 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.957945 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.957948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.959052 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:50.959063 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.959875 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.959879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:50.959999 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.960027 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 02:45:50.960169 (+ 142us) client_negotiation.cc:240] Negotiation successful
0806 02:45:50.960247 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3467,"mutex_wait_us":350,"thread_start_us":114,"threads_started":1}
I20260806 02:45:50.960417 12796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:50.952641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:47597 (local address 127.1.183.130:33151)
0806 02:45:50.952920 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:50.952924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:50.956244 (+ 3320us) server_negotiation.cc:407] Connection header received
0806 02:45:50.956454 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:50.956458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:50.956505 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:50.956553 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:50.957014 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.957705 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.959180 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:50.959739 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:50.959845 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:50.959881 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:50.960220 (+ 339us) server_negotiation.cc:299] Negotiation successful
0806 02:45:50.960302 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210}
I20260806 02:45:50.961035 12473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34a2905691c8446cb9774cc315d77934" candidate_uuid: "fff12eae637c41ebabb8689441a6dbe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2" is_pre_election: true
I20260806 02:45:50.961728 12183 leader_election.cc:302] T 34a2905691c8446cb9774cc315d77934 P fff12eae637c41ebabb8689441a6dbe5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fff12eae637c41ebabb8689441a6dbe5; no voters: 5e8dc07e3a1f4de49b0e6a5bec6878f2, e2cbd8619d9048b5b56e4d9ea417288e
I20260806 02:45:50.962047 12805 raft_consensus.cc:2726] T 34a2905691c8446cb9774cc315d77934 P fff12eae637c41ebabb8689441a6dbe5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 02:45:50.964620 10983 catalog_manager.cc:5495] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e8dc07e3a1f4de49b0e6a5bec6878f2 (127.1.183.130). New cstate: current_term: 1 leader_uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff12eae637c41ebabb8689441a6dbe5" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 34391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2cbd8619d9048b5b56e4d9ea417288e" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 39727 } health_report { overall_health: UNKNOWN } } }
W20260806 02:45:50.967101 12557 tablet.cc:2369] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:45:50.995800 12339 tablet.cc:2369] T 34a2905691c8446cb9774cc315d77934 P fff12eae637c41ebabb8689441a6dbe5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:45:51.006831 12796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:51.002749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36932 (local address 127.1.183.130:33151)
0806 02:45:51.003002 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:51.003007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:51.003022 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:45:51.003166 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:51.003168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:51.003228 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:51.003346 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:51.003603 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:51.004135 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:51.005133 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:51.005423 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:51.005533 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:51.005887 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:51.006347 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:51.006488 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:51.006603 (+ 115us) server_negotiation.cc:299] Negotiation successful
0806 02:45:51.006664 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260806 02:45:51.013465 12688 raft_consensus.cc:1261] T 34a2905691c8446cb9774cc315d77934 P e2cbd8619d9048b5b56e4d9ea417288e [term 1 FOLLOWER]: Refusing update from remote peer 5e8dc07e3a1f4de49b0e6a5bec6878f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:51.013500 12281 raft_consensus.cc:1261] T 34a2905691c8446cb9774cc315d77934 P fff12eae637c41ebabb8689441a6dbe5 [term 1 FOLLOWER]: Refusing update from remote peer 5e8dc07e3a1f4de49b0e6a5bec6878f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:51.014312 12813 consensus_queue.cc:1035] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cbd8619d9048b5b56e4d9ea417288e" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 39727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:51.014515 12810 consensus_queue.cc:1035] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fff12eae637c41ebabb8689441a6dbe5" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:51.023360 12818 mvcc.cc:205] Tried to move back new op lower bound from 7315391901743267840 to 7315391901536735232. Current Snapshot: MvccSnapshot[applied={T|T < 7315391901743267840}]
I20260806 02:45:51.025879 12820 mvcc.cc:205] Tried to move back new op lower bound from 7315391901743267840 to 7315391901536735232. Current Snapshot: MvccSnapshot[applied={T|T < 7315391901743267840}]
I20260806 02:45:51.034974 10982 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:51.035138 10982 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:51.041565 10982 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5975de0199eb4bc1a2ac5d5607e1928c: Sending DeleteTablet for 3 replicas of tablet 34a2905691c8446cb9774cc315d77934
I20260806 02:45:51.042531 12439 tablet_service.cc:1514] Processing DeleteTablet for tablet 34a2905691c8446cb9774cc315d77934 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:51 UTC) from {username='oryx', principal='oryx/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:36926
I20260806 02:45:51.043116 12248 tablet_service.cc:1514] Processing DeleteTablet for tablet 34a2905691c8446cb9774cc315d77934 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:51 UTC) from {username='oryx', principal='oryx/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:35854
I20260806 02:45:51.043226 12657 tablet_service.cc:1514] Processing DeleteTablet for tablet 34a2905691c8446cb9774cc315d77934 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:51 UTC) from {username='oryx', principal='oryx/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:54994
I20260806 02:45:51.043895 12833 tablet_replica.cc:330] T 34a2905691c8446cb9774cc315d77934 P fff12eae637c41ebabb8689441a6dbe5: stopping tablet replica
I20260806 02:45:51.043907 12831 tablet_replica.cc:330] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2: stopping tablet replica
I20260806 02:45:51.044335 12833 raft_consensus.cc:2229] T 34a2905691c8446cb9774cc315d77934 P fff12eae637c41ebabb8689441a6dbe5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:51.044337 12831 raft_consensus.cc:2229] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:45:51.044771 12833 raft_consensus.cc:2258] T 34a2905691c8446cb9774cc315d77934 P fff12eae637c41ebabb8689441a6dbe5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:51.045022 12831 raft_consensus.cc:2258] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:51.045944 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 10940
I20260806 02:45:51.053454 12833 ts_tablet_manager.cc:1905] T 34a2905691c8446cb9774cc315d77934 P fff12eae637c41ebabb8689441a6dbe5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:51.053467 12831 ts_tablet_manager.cc:1905] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:51.056723 12833 ts_tablet_manager.cc:1918] T 34a2905691c8446cb9774cc315d77934 P fff12eae637c41ebabb8689441a6dbe5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:45:51.057049 12833 log.cc:1196] T 34a2905691c8446cb9774cc315d77934 P fff12eae637c41ebabb8689441a6dbe5: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-0/wal/wals/34a2905691c8446cb9774cc315d77934
I20260806 02:45:51.057492 12833 ts_tablet_manager.cc:1939] T 34a2905691c8446cb9774cc315d77934 P fff12eae637c41ebabb8689441a6dbe5: Deleting consensus metadata
I20260806 02:45:51.057478 12831 ts_tablet_manager.cc:1918] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:45:51.057595 12831 log.cc:1196] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-1/wal/wals/34a2905691c8446cb9774cc315d77934
I20260806 02:45:51.057994 12831 ts_tablet_manager.cc:1939] T 34a2905691c8446cb9774cc315d77934 P 5e8dc07e3a1f4de49b0e6a5bec6878f2: Deleting consensus metadata
W20260806 02:45:51.063701 12368 connection.cc:535] server connection from 127.0.0.1:36926 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260806 02:45:51.064373 12181 connection.cc:535] server connection from 127.0.0.1:35854 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 02:45:51.065086 12834 tablet_replica.cc:330] T 34a2905691c8446cb9774cc315d77934 P e2cbd8619d9048b5b56e4d9ea417288e: stopping tablet replica
I20260806 02:45:51.065563 12834 raft_consensus.cc:2229] T 34a2905691c8446cb9774cc315d77934 P e2cbd8619d9048b5b56e4d9ea417288e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:51.065946 12834 raft_consensus.cc:2258] T 34a2905691c8446cb9774cc315d77934 P e2cbd8619d9048b5b56e4d9ea417288e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:51.067673 12834 ts_tablet_manager.cc:1905] T 34a2905691c8446cb9774cc315d77934 P e2cbd8619d9048b5b56e4d9ea417288e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:51.070988 12834 ts_tablet_manager.cc:1918] T 34a2905691c8446cb9774cc315d77934 P e2cbd8619d9048b5b56e4d9ea417288e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:45:51.071174 12834 log.cc:1196] T 34a2905691c8446cb9774cc315d77934 P e2cbd8619d9048b5b56e4d9ea417288e: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-2/wal/wals/34a2905691c8446cb9774cc315d77934
I20260806 02:45:51.071596 12834 ts_tablet_manager.cc:1939] T 34a2905691c8446cb9774cc315d77934 P e2cbd8619d9048b5b56e4d9ea417288e: Deleting consensus metadata
W20260806 02:45:51.072973 12587 connection.cc:441] server connection from 127.0.0.1:54994 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54994 (request call id 1) could send its response
I20260806 02:45:51.074662 12777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:51.044583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51228 (local address 127.1.183.189:40231)
0806 02:45:51.044710 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:51.044715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:51.045481 (+ 766us) server_negotiation.cc:407] Connection header received
0806 02:45:51.059709 (+ 14228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:51.059715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:51.059800 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:51.059881 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:51.061203 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:51.062175 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:51.063634 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:51.063993 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:51.064096 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:51.070505 (+ 6409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:51.070510 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:51.070513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:51.070550 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:51.072670 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:51.073197 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:51.073202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:51.073203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:51.073296 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:51.073605 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:51.073608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:51.073610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:51.073842 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:51.073978 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:51.074310 (+ 332us) server_negotiation.cc:299] Negotiation successful
0806 02:45:51.074457 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 02:45:51.081454 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:51.069152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37768 (local address 127.1.183.188:35427)
0806 02:45:51.069312 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:51.069317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:51.071114 (+ 1797us) server_negotiation.cc:407] Connection header received
0806 02:45:51.071346 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:51.071350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:51.071465 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:51.071529 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:51.072646 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:51.073464 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:51.074644 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:51.075004 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:51.075124 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:51.077023 (+ 1899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:51.077027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:51.077030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:51.077059 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:51.079045 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:51.079880 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:51.079884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:51.079886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:51.079982 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:51.080434 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:51.080438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:51.080440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:51.080660 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:51.080771 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:51.081131 (+ 360us) server_negotiation.cc:299] Negotiation successful
0806 02:45:51.081266 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:45:51.085362 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:51.070015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51244 (local address 127.1.183.189:40231)
0806 02:45:51.070208 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:51.070212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:51.074855 (+ 4643us) server_negotiation.cc:407] Connection header received
0806 02:45:51.075025 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:51.075029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:51.075080 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:51.075134 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:51.076577 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:51.077327 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:51.078605 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:51.078939 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:51.079013 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:51.081394 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:51.081398 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:51.081401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:51.081432 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:51.083250 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:51.083805 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:51.083809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:51.083811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:51.083915 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:51.084344 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:51.084348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:51.084350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:51.084635 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:51.084778 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:51.085054 (+ 276us) server_negotiation.cc:299] Negotiation successful
0806 02:45:51.085188 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 02:45:51.085397 12773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:51.044455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37756 (local address 127.1.183.188:35427)
0806 02:45:51.044573 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:51.044578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:51.076718 (+ 32140us) server_negotiation.cc:407] Connection header received
0806 02:45:51.076947 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:51.076950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:51.077007 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:51.077056 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:51.077849 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:51.078617 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:51.079703 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:51.080033 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:51.080124 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:51.081707 (+ 1583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:51.081712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:51.081715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:51.081744 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:51.083485 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:51.083948 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:51.083952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:51.083954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:51.084046 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:51.084461 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:51.084465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:51.084466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:51.084656 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:51.084776 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:51.085134 (+ 358us) server_negotiation.cc:299] Negotiation successful
0806 02:45:51.085261 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 02:45:52.029405 12849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:52.028825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.1.183.129:58811)
0806 02:45:52.029150 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:52.029268 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":95,"threads_started":1}
I20260806 02:45:52.030603 12853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:52.030083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.1.183.130:33007)
0806 02:45:52.030328 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:52.030493 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":99,"threads_started":1}
W20260806 02:45:52.030835 12332 heartbeater.cc:644] Failed to heartbeat to 127.1.183.190:39091 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
W20260806 02:45:52.030902 12552 heartbeater.cc:644] Failed to heartbeat to 127.1.183.190:39091 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
I20260806 02:45:52.031548 12851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:52.030026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.1.183.131:40723)
0806 02:45:52.031339 (+ 1313us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:52.031437 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1219,"thread_start_us":84,"threads_started":1}
W20260806 02:45:52.031859 12755 heartbeater.cc:644] Failed to heartbeat to 127.1.183.190:39091 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
I20260806 02:45:52.541077 12855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5975de0199eb4bc1a2ac5d5607e1928c)
I20260806 02:45:52.541169 12856 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5975de0199eb4bc1a2ac5d5607e1928c)
I20260806 02:45:52.541364 12855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: false } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: false } }
I20260806 02:45:52.541399 12856 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: false } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: false } }
I20260806 02:45:52.542518 12856 leader_election.cc:288] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5975de0199eb4bc1a2ac5d5607e1928c (127.1.183.190:39091), 938ea1eb1f524f2eacb24a0224ce4694 (127.1.183.188:35427)
I20260806 02:45:52.542907 12855 leader_election.cc:288] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5975de0199eb4bc1a2ac5d5607e1928c (127.1.183.190:39091), fb3e8a85dcf24fb7b146b28dd2053249 (127.1.183.189:40231)
I20260806 02:45:52.543186 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:52.542684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.0.0.1:43468)
0806 02:45:52.542963 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:52.543053 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":111,"threads_started":1}
I20260806 02:45:52.543422 12857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:52.542682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.0.0.1:43474)
0806 02:45:52.543205 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:52.543306 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":81,"threads_started":1}
W20260806 02:45:52.543553 11809 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
W20260806 02:45:52.543736 12070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
I20260806 02:45:52.544963 12858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:52.544673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.0.0.1:43484)
0806 02:45:52.544840 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:52.544892 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260806 02:45:52.544991 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:52.544654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.0.0.1:43478)
0806 02:45:52.544860 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:52.544916 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260806 02:45:52.545065 12070 leader_election.cc:334] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5975de0199eb4bc1a2ac5d5607e1928c (127.1.183.190:39091): Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
W20260806 02:45:52.545248 11809 leader_election.cc:334] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5975de0199eb4bc1a2ac5d5607e1928c (127.1.183.190:39091): Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
I20260806 02:45:52.547945 12857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:52.542711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:40231 (local address 127.0.0.1:46930)
0806 02:45:52.543574 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:52.543648 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:52.543752 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:52.544029 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:52.544032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:52.544054 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:52.544199 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:52.544207 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:52.545283 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:52.545288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:52.546577 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:52.546588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:52.547453 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:52.547455 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:52.547599 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:52.547620 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 02:45:52.547722 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 02:45:52.547796 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"mutex_wait_us":42,"thread_start_us":72,"threads_started":1}
I20260806 02:45:52.547968 12862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:52.543315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46930 (local address 127.1.183.189:40231)
0806 02:45:52.543782 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:52.543789 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:52.543818 (+ 29us) server_negotiation.cc:407] Connection header received
0806 02:45:52.543868 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:52.543871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:52.543916 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:52.544009 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:52.544349 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:52.545135 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:52.546713 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:52.547315 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:52.547440 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:52.547510 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:52.547765 (+ 255us) server_negotiation.cc:299] Negotiation successful
0806 02:45:52.547843 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":393,"thread_start_us":61,"threads_started":1}
I20260806 02:45:52.548189 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:52.543088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:35427 (local address 127.0.0.1:43160)
0806 02:45:52.543396 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:52.543421 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:52.543536 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:52.544286 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:52.544291 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:52.544313 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:52.544501 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:52.544515 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:52.545582 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:52.545586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:52.546769 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:52.546779 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:52.547701 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:52.547707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:52.547832 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:52.547858 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 02:45:52.547931 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 02:45:52.548016 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":51,"thread_start_us":71,"threads_started":1}
I20260806 02:45:52.548243 12863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:52.543492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43160 (local address 127.1.183.188:35427)
0806 02:45:52.543994 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:52.543999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:52.544013 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:45:52.544057 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:52.544060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:52.544177 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:52.544260 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:52.544652 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:52.545457 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:52.546916 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:52.547584 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:52.547663 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:52.547737 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:52.547968 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 02:45:52.548057 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":426,"spinlock_wait_cycles":768,"thread_start_us":68,"threads_started":1}
I20260806 02:45:52.548532 11843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "938ea1eb1f524f2eacb24a0224ce4694" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" is_pre_election: true
I20260806 02:45:52.548723 12093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "938ea1eb1f524f2eacb24a0224ce4694" is_pre_election: true
I20260806 02:45:52.548789 11843 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 938ea1eb1f524f2eacb24a0224ce4694 in term 1.
I20260806 02:45:52.548909 12093 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb3e8a85dcf24fb7b146b28dd2053249 in term 1.
I20260806 02:45:52.549239 11809 leader_election.cc:302] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 938ea1eb1f524f2eacb24a0224ce4694, fb3e8a85dcf24fb7b146b28dd2053249; no voters: 5975de0199eb4bc1a2ac5d5607e1928c
I20260806 02:45:52.549240 12064 leader_election.cc:302] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 938ea1eb1f524f2eacb24a0224ce4694, fb3e8a85dcf24fb7b146b28dd2053249; no voters: 5975de0199eb4bc1a2ac5d5607e1928c
I20260806 02:45:52.549484 12855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 02:45:52.549556 12855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5975de0199eb4bc1a2ac5d5607e1928c)
I20260806 02:45:52.549619 12855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:45:52.549655 12856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 02:45:52.549718 12856 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5975de0199eb4bc1a2ac5d5607e1928c)
I20260806 02:45:52.549774 12856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:45:52.551000 12856 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: false } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: false } }
I20260806 02:45:52.551000 12855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: false } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: false } }
I20260806 02:45:52.551409 12855 leader_election.cc:288] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [CANDIDATE]: Term 2 election: Requested vote from peers 5975de0199eb4bc1a2ac5d5607e1928c (127.1.183.190:39091), fb3e8a85dcf24fb7b146b28dd2053249 (127.1.183.189:40231)
I20260806 02:45:52.551401 12856 leader_election.cc:288] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [CANDIDATE]: Term 2 election: Requested vote from peers 5975de0199eb4bc1a2ac5d5607e1928c (127.1.183.190:39091), 938ea1eb1f524f2eacb24a0224ce4694 (127.1.183.188:35427)
I20260806 02:45:52.551797 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:52.551535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.0.0.1:43494)
0806 02:45:52.551671 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:52.551728 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 02:45:52.551829 12857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:52.551535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.0.0.1:43496)
0806 02:45:52.551729 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:52.551774 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260806 02:45:52.551893 11843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "938ea1eb1f524f2eacb24a0224ce4694" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fb3e8a85dcf24fb7b146b28dd2053249"
I20260806 02:45:52.552068 11843 raft_consensus.cc:2379] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 938ea1eb1f524f2eacb24a0224ce4694 in current term 2: Already voted for candidate fb3e8a85dcf24fb7b146b28dd2053249 in this term.
I20260806 02:45:52.552084 12093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "938ea1eb1f524f2eacb24a0224ce4694"
I20260806 02:45:52.552260 12093 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fb3e8a85dcf24fb7b146b28dd2053249 in current term 2: Already voted for candidate 938ea1eb1f524f2eacb24a0224ce4694 in this term.
I20260806 02:45:52.552770 12857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:52.552561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.0.0.1:43522)
0806 02:45:52.552667 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:52.552722 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260806 02:45:52.552891 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:52.552426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.0.0.1:43508)
0806 02:45:52.552778 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:52.552834 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286}
W20260806 02:45:52.552882 12070 leader_election.cc:334] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5975de0199eb4bc1a2ac5d5607e1928c (127.1.183.190:39091): Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
I20260806 02:45:52.552938 12070 leader_election.cc:302] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 938ea1eb1f524f2eacb24a0224ce4694; no voters: 5975de0199eb4bc1a2ac5d5607e1928c, fb3e8a85dcf24fb7b146b28dd2053249
W20260806 02:45:52.553009 11809 leader_election.cc:334] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5975de0199eb4bc1a2ac5d5607e1928c (127.1.183.190:39091): Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
I20260806 02:45:52.553069 11809 leader_election.cc:302] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb3e8a85dcf24fb7b146b28dd2053249; no voters: 5975de0199eb4bc1a2ac5d5607e1928c, 938ea1eb1f524f2eacb24a0224ce4694
I20260806 02:45:52.553077 12855 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 02:45:52.553200 12856 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 02:45:53.032428 12867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:53.031613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.1.183.129:35063)
0806 02:45:53.032203 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:53.032293 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":127,"threads_started":1}
I20260806 02:45:53.032625 12869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:53.032012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.1.183.130:42215)
0806 02:45:53.032442 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:53.032512 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":75,"threads_started":1}
I20260806 02:45:53.033444 12870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:53.033011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.1.183.131:42937)
0806 02:45:53.033291 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:53.033361 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":87,"threads_started":1}
I20260806 02:45:54.033903 12872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:54.033243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.1.183.130:41577)
0806 02:45:54.033680 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:54.033766 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":88,"threads_started":1}
I20260806 02:45:54.034754 12876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:54.034074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.1.183.131:39433)
0806 02:45:54.034584 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:54.034650 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":90,"threads_started":1}
I20260806 02:45:54.034754 12873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:54.033252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.1.183.129:44831)
0806 02:45:54.034584 (+ 1332us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:54.034650 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1263,"thread_start_us":85,"threads_started":1}
I20260806 02:45:54.432667 12877 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:45:54.432863 12877 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: false } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: false } }
I20260806 02:45:54.433375 12877 leader_election.cc:288] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5975de0199eb4bc1a2ac5d5607e1928c (127.1.183.190:39091), 938ea1eb1f524f2eacb24a0224ce4694 (127.1.183.188:35427)
I20260806 02:45:54.434079 12878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:54.433568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.0.0.1:43788)
0806 02:45:54.433868 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:54.433950 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":87,"threads_started":1}
I20260806 02:45:54.434262 12093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "938ea1eb1f524f2eacb24a0224ce4694" is_pre_election: true
I20260806 02:45:54.434458 12093 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb3e8a85dcf24fb7b146b28dd2053249 in term 2.
I20260806 02:45:54.434926 11809 leader_election.cc:302] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 938ea1eb1f524f2eacb24a0224ce4694, fb3e8a85dcf24fb7b146b28dd2053249; no voters:
I20260806 02:45:54.435017 12878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:54.434788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.0.0.1:43794)
0806 02:45:54.434908 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:54.434959 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 02:45:54.435169 11809 leader_election.cc:334] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5975de0199eb4bc1a2ac5d5607e1928c (127.1.183.190:39091): Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
I20260806 02:45:54.435174 12877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 02:45:54.435267 12877 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:45:54.435308 12877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [term 2 FOLLOWER]: Advancing to term 3
I20260806 02:45:54.436535 12877 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: false } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: false } }
I20260806 02:45:54.436899 12877 leader_election.cc:288] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [CANDIDATE]: Term 3 election: Requested vote from peers 5975de0199eb4bc1a2ac5d5607e1928c (127.1.183.190:39091), 938ea1eb1f524f2eacb24a0224ce4694 (127.1.183.188:35427)
I20260806 02:45:54.437307 12878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:54.437006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.0.0.1:43806)
0806 02:45:54.437115 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:54.437174 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 02:45:54.437513 12093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "938ea1eb1f524f2eacb24a0224ce4694"
I20260806 02:45:54.437685 12093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [term 2 FOLLOWER]: Advancing to term 3
I20260806 02:45:54.438091 12878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:54.437872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.0.0.1:43814)
0806 02:45:54.437982 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:54.438035 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260806 02:45:54.438251 11809 leader_election.cc:334] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 5975de0199eb4bc1a2ac5d5607e1928c (127.1.183.190:39091): Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
I20260806 02:45:54.438933 12093 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb3e8a85dcf24fb7b146b28dd2053249 in term 3.
I20260806 02:45:54.439227 11809 leader_election.cc:302] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 938ea1eb1f524f2eacb24a0224ce4694, fb3e8a85dcf24fb7b146b28dd2053249; no voters: 5975de0199eb4bc1a2ac5d5607e1928c
I20260806 02:45:54.439414 12877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 02:45:54.439615 12877 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [term 3 LEADER]: Becoming Leader. State: Replica: fb3e8a85dcf24fb7b146b28dd2053249, State: Running, Role: LEADER
I20260806 02:45:54.439886 12877 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: false } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: false } }
I20260806 02:45:54.441007 12885 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb3e8a85dcf24fb7b146b28dd2053249. Latest consensus state: current_term: 3 leader_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: false } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: false } } }
I20260806 02:45:54.441147 12885 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:54.441466 12888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:54.444299 12888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:54.444778 12888 catalog_manager.cc:1259] Loaded cluster ID: 07ac2c30b95046aa9ad9b2c2891f35ca
I20260806 02:45:54.444823 12888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:54.445420 12888 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:54.445838 12888 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249: Loaded TSK: 0
I20260806 02:45:54.446571 12888 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:45:54.478708 11827 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:45:54.482203 11827 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:45:54.485714 12878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:54.485425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.0.0.1:43828)
0806 02:45:54.485570 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:54.485635 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 02:45:54.485729 12093 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [term 3 FOLLOWER]: Refusing update from remote peer fb3e8a85dcf24fb7b146b28dd2053249: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260806 02:45:54.486264 12877 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [LEADER]: Connected to new peer: Peer: permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260806 02:45:54.486696 12878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:54.486372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.0.0.1:43834)
0806 02:45:54.486506 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:54.486598 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260806 02:45:54.486886 11809 consensus_peers.cc:487] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 -> Peer 5975de0199eb4bc1a2ac5d5607e1928c (127.1.183.190:39091): Couldn't send request to peer 5975de0199eb4bc1a2ac5d5607e1928c. Status: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 02:45:54.488257 12891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb3e8a85dcf24fb7b146b28dd2053249. Latest consensus state: current_term: 3 leader_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: false } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: false } } }
I20260806 02:45:54.488404 12891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:45:54.489607 12885 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: false } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: false } } }
I20260806 02:45:54.489748 12885 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249 [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:54.490029 12891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "5975de0199eb4bc1a2ac5d5607e1928c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39091 } } peers { permanent_uuid: "fb3e8a85dcf24fb7b146b28dd2053249" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 40231 } attrs { promote: false } } peers { permanent_uuid: "938ea1eb1f524f2eacb24a0224ce4694" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35427 } attrs { promote: false } } }
I20260806 02:45:54.490233 12891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 938ea1eb1f524f2eacb24a0224ce4694 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:45:54.490408 12893 mvcc.cc:205] Tried to move back new op lower bound from 7315391915968131072 to 7315391915788333056. Current Snapshot: MvccSnapshot[applied={T|T < 7315391915968131072 or (T in {7315391915968131072})}]
I20260806 02:45:54.501044 12878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:54.496378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:39727 (local address 127.0.0.1:42008)
0806 02:45:54.496867 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:54.496879 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:54.497014 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:54.497570 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:54.497576 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:54.497600 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:54.497734 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:54.497741 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:54.498611 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:54.498616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:54.499739 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:54.499749 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:54.500666 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:54.500670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:54.500760 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:54.500789 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 02:45:54.500837 (+ 48us) client_negotiation.cc:240] Negotiation successful
0806 02:45:54.500903 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429}
I20260806 02:45:54.501542 12898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:54.496942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:34391 (local address 127.0.0.1:49932)
0806 02:45:54.497381 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:54.497396 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:54.497482 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:54.498531 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:54.498534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:54.498551 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:54.498643 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:54.498649 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:54.499392 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:54.499397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:54.500307 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:54.500317 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:54.501141 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:54.501146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:54.501261 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:54.501284 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 02:45:54.501329 (+ 45us) client_negotiation.cc:240] Negotiation successful
0806 02:45:54.501395 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":114,"threads_started":1}
I20260806 02:45:54.501665 12897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:54.496488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42008 (local address 127.1.183.131:39727)
0806 02:45:54.496800 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:54.496805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:54.496985 (+ 180us) server_negotiation.cc:407] Connection header received
0806 02:45:54.497116 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:54.497119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:54.497176 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:54.497844 (+ 668us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:54.497955 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:54.498495 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:54.499875 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:54.500534 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:54.501322 (+ 788us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:54.501391 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:54.501441 (+ 50us) server_negotiation.cc:299] Negotiation successful
0806 02:45:54.501535 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":106,"threads_started":1}
I20260806 02:45:54.502260 12657 tablet_service.cc:1467] Processing CreateTablet for tablet 1487a2cb66e64f338dbfb7bbccf3eda3 (DEFAULT_TABLE table=test-table [id=ee6846dbebf64bb496e13bebb5e43869]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:54.502374 12900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:54.497099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49932 (local address 127.1.183.129:34391)
0806 02:45:54.498290 (+ 1191us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:54.498293 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:54.498306 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:45:54.498351 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:54.498353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:54.498391 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:54.498512 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:54.498770 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:54.499280 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:54.500446 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:54.501013 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:54.501533 (+ 520us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:54.502099 (+ 566us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:54.502139 (+ 40us) server_negotiation.cc:299] Negotiation successful
0806 02:45:54.502242 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1109,"thread_start_us":82,"threads_started":1}
I20260806 02:45:54.502607 12657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1487a2cb66e64f338dbfb7bbccf3eda3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:54.502844 12248 tablet_service.cc:1467] Processing CreateTablet for tablet 1487a2cb66e64f338dbfb7bbccf3eda3 (DEFAULT_TABLE table=test-table [id=ee6846dbebf64bb496e13bebb5e43869]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:54.503124 12248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1487a2cb66e64f338dbfb7bbccf3eda3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:54.504251 12901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:54.497269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:33151 (local address 127.0.0.1:56494)
0806 02:45:54.497525 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:54.497540 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:54.497652 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:54.499186 (+ 1534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:54.499189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:54.499206 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:54.499283 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:54.499289 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:54.500313 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:54.500317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:54.501400 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:54.501412 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:54.502246 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:54.502249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:54.502337 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:54.502360 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 02:45:54.502438 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 02:45:54.502518 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":92,"threads_started":1}
I20260806 02:45:54.504763 12902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:54.497378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56494 (local address 127.1.183.130:33151)
0806 02:45:54.498766 (+ 1388us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:54.498770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:54.498784 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:45:54.498833 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:54.498836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:54.498881 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:54.498976 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:54.499437 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:54.500192 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:54.501535 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:54.502117 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:54.502225 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:54.502284 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:54.504248 (+ 1964us) server_negotiation.cc:299] Negotiation successful
0806 02:45:54.504633 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1303,"thread_start_us":87,"threads_started":1}
I20260806 02:45:54.505496 12439 tablet_service.cc:1467] Processing CreateTablet for tablet 1487a2cb66e64f338dbfb7bbccf3eda3 (DEFAULT_TABLE table=test-table [id=ee6846dbebf64bb496e13bebb5e43869]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:54.505717 12903 tablet_bootstrap.cc:492] T 1487a2cb66e64f338dbfb7bbccf3eda3 P e2cbd8619d9048b5b56e4d9ea417288e: Bootstrap starting.
I20260806 02:45:54.505756 12439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1487a2cb66e64f338dbfb7bbccf3eda3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:54.505973 12904 tablet_bootstrap.cc:492] T 1487a2cb66e64f338dbfb7bbccf3eda3 P fff12eae637c41ebabb8689441a6dbe5: Bootstrap starting.
I20260806 02:45:54.506677 12903 tablet_bootstrap.cc:654] T 1487a2cb66e64f338dbfb7bbccf3eda3 P e2cbd8619d9048b5b56e4d9ea417288e: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:54.506822 12904 tablet_bootstrap.cc:654] T 1487a2cb66e64f338dbfb7bbccf3eda3 P fff12eae637c41ebabb8689441a6dbe5: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:54.508038 12903 tablet_bootstrap.cc:492] T 1487a2cb66e64f338dbfb7bbccf3eda3 P e2cbd8619d9048b5b56e4d9ea417288e: No bootstrap required, opened a new log
I20260806 02:45:54.508136 12903 ts_tablet_manager.cc:1397] T 1487a2cb66e64f338dbfb7bbccf3eda3 P e2cbd8619d9048b5b56e4d9ea417288e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:45:54.508136 12907 tablet_bootstrap.cc:492] T 1487a2cb66e64f338dbfb7bbccf3eda3 P 5e8dc07e3a1f4de49b0e6a5bec6878f2: Bootstrap starting.
I20260806 02:45:54.508630 12903 raft_consensus.cc:348] T 1487a2cb66e64f338dbfb7bbccf3eda3 P e2cbd8619d9048b5b56e4d9ea417288e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cbd8619d9048b5b56e4d9ea417288e" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 39727 } } peers { permanent_uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33151 } } peers { permanent_uuid: "fff12eae637c41ebabb8689441a6dbe5" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 34391 } }
I20260806 02:45:54.508770 12903 raft_consensus.cc:374] T 1487a2cb66e64f338dbfb7bbccf3eda3 P e2cbd8619d9048b5b56e4d9ea417288e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:54.508808 12903 raft_consensus.cc:729] T 1487a2cb66e64f338dbfb7bbccf3eda3 P e2cbd8619d9048b5b56e4d9ea417288e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2cbd8619d9048b5b56e4d9ea417288e, State: Initialized, Role: FOLLOWER
I20260806 02:45:54.508953 12904 tablet_bootstrap.cc:492] T 1487a2cb66e64f338dbfb7bbccf3eda3 P fff12eae637c41ebabb8689441a6dbe5: No bootstrap required, opened a new log
I20260806 02:45:54.508941 12903 consensus_queue.cc:260] T 1487a2cb66e64f338dbfb7bbccf3eda3 P e2cbd8619d9048b5b56e4d9ea417288e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cbd8619d9048b5b56e4d9ea417288e" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 39727 } } peers { permanent_uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33151 } } peers { permanent_uuid: "fff12eae637c41ebabb8689441a6dbe5" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 34391 } }
I20260806 02:45:54.509025 12904 ts_tablet_manager.cc:1397] T 1487a2cb66e64f338dbfb7bbccf3eda3 P fff12eae637c41ebabb8689441a6dbe5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260806 02:45:54.509052 12907 tablet_bootstrap.cc:654] T 1487a2cb66e64f338dbfb7bbccf3eda3 P 5e8dc07e3a1f4de49b0e6a5bec6878f2: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:54.509327 12903 ts_tablet_manager.cc:1428] T 1487a2cb66e64f338dbfb7bbccf3eda3 P e2cbd8619d9048b5b56e4d9ea417288e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 02:45:54.509481 12904 raft_consensus.cc:348] T 1487a2cb66e64f338dbfb7bbccf3eda3 P fff12eae637c41ebabb8689441a6dbe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cbd8619d9048b5b56e4d9ea417288e" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 39727 } } peers { permanent_uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33151 } } peers { permanent_uuid: "fff12eae637c41ebabb8689441a6dbe5" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 34391 } }
I20260806 02:45:54.509596 12904 raft_consensus.cc:374] T 1487a2cb66e64f338dbfb7bbccf3eda3 P fff12eae637c41ebabb8689441a6dbe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:54.509642 12904 raft_consensus.cc:729] T 1487a2cb66e64f338dbfb7bbccf3eda3 P fff12eae637c41ebabb8689441a6dbe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fff12eae637c41ebabb8689441a6dbe5, State: Initialized, Role: FOLLOWER
I20260806 02:45:54.509794 12904 consensus_queue.cc:260] T 1487a2cb66e64f338dbfb7bbccf3eda3 P fff12eae637c41ebabb8689441a6dbe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cbd8619d9048b5b56e4d9ea417288e" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 39727 } } peers { permanent_uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33151 } } peers { permanent_uuid: "fff12eae637c41ebabb8689441a6dbe5" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 34391 } }
I20260806 02:45:54.510257 12876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:54.509856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.1.183.131:49755)
0806 02:45:54.510119 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:54.510184 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189}
I20260806 02:45:54.510749 12873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:54.510474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.1.183.129:53387)
0806 02:45:54.510629 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:54.510685 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260806 02:45:54.511375 12904 ts_tablet_manager.cc:1428] T 1487a2cb66e64f338dbfb7bbccf3eda3 P fff12eae637c41ebabb8689441a6dbe5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:45:54.512080 12873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:54.511787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.1.183.129:37923)
0806 02:45:54.511953 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:54.512008 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260806 02:45:54.512910 12335 heartbeater.cc:503] Master 127.1.183.189:40231 was elected leader, sending a full tablet report...
I20260806 02:45:54.513295 12907 tablet_bootstrap.cc:492] T 1487a2cb66e64f338dbfb7bbccf3eda3 P 5e8dc07e3a1f4de49b0e6a5bec6878f2: No bootstrap required, opened a new log
I20260806 02:45:54.513388 12907 ts_tablet_manager.cc:1397] T 1487a2cb66e64f338dbfb7bbccf3eda3 P 5e8dc07e3a1f4de49b0e6a5bec6878f2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260806 02:45:54.513840 12907 raft_consensus.cc:348] T 1487a2cb66e64f338dbfb7bbccf3eda3 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cbd8619d9048b5b56e4d9ea417288e" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 39727 } } peers { permanent_uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33151 } } peers { permanent_uuid: "fff12eae637c41ebabb8689441a6dbe5" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 34391 } }
I20260806 02:45:54.513964 12907 raft_consensus.cc:374] T 1487a2cb66e64f338dbfb7bbccf3eda3 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:54.514011 12907 raft_consensus.cc:729] T 1487a2cb66e64f338dbfb7bbccf3eda3 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8dc07e3a1f4de49b0e6a5bec6878f2, State: Initialized, Role: FOLLOWER
I20260806 02:45:54.514147 12907 consensus_queue.cc:260] T 1487a2cb66e64f338dbfb7bbccf3eda3 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cbd8619d9048b5b56e4d9ea417288e" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 39727 } } peers { permanent_uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33151 } } peers { permanent_uuid: "fff12eae637c41ebabb8689441a6dbe5" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 34391 } }
I20260806 02:45:54.514788 12907 ts_tablet_manager.cc:1428] T 1487a2cb66e64f338dbfb7bbccf3eda3 P 5e8dc07e3a1f4de49b0e6a5bec6878f2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 02:45:54.515103 12872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:54.514829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.1.183.130:55459)
0806 02:45:54.514977 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:54.515032 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260806 02:45:54.515857 12872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:54.515592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.1.183.130:47055)
0806 02:45:54.515714 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:54.515781 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 02:45:54.516052 12556 heartbeater.cc:503] Master 127.1.183.189:40231 was elected leader, sending a full tablet report...
I20260806 02:45:54.563865 12914 raft_consensus.cc:482] T 1487a2cb66e64f338dbfb7bbccf3eda3 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:45:54.564050 12914 raft_consensus.cc:504] T 1487a2cb66e64f338dbfb7bbccf3eda3 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cbd8619d9048b5b56e4d9ea417288e" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 39727 } } peers { permanent_uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33151 } } peers { permanent_uuid: "fff12eae637c41ebabb8689441a6dbe5" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 34391 } }
I20260806 02:45:54.564610 12914 leader_election.cc:288] T 1487a2cb66e64f338dbfb7bbccf3eda3 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2cbd8619d9048b5b56e4d9ea417288e (127.1.183.131:39727), fff12eae637c41ebabb8689441a6dbe5 (127.1.183.129:34391)
I20260806 02:45:54.565186 12281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1487a2cb66e64f338dbfb7bbccf3eda3" candidate_uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fff12eae637c41ebabb8689441a6dbe5" is_pre_election: true
I20260806 02:45:54.565186 12688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1487a2cb66e64f338dbfb7bbccf3eda3" candidate_uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2cbd8619d9048b5b56e4d9ea417288e" is_pre_election: true
I20260806 02:45:54.565376 12281 raft_consensus.cc:2454] T 1487a2cb66e64f338dbfb7bbccf3eda3 P fff12eae637c41ebabb8689441a6dbe5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8dc07e3a1f4de49b0e6a5bec6878f2 in term 0.
I20260806 02:45:54.565376 12688 raft_consensus.cc:2454] T 1487a2cb66e64f338dbfb7bbccf3eda3 P e2cbd8619d9048b5b56e4d9ea417288e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8dc07e3a1f4de49b0e6a5bec6878f2 in term 0.
I20260806 02:45:54.565785 12367 leader_election.cc:302] T 1487a2cb66e64f338dbfb7bbccf3eda3 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8dc07e3a1f4de49b0e6a5bec6878f2, e2cbd8619d9048b5b56e4d9ea417288e; no voters:
I20260806 02:45:54.566027 12914 raft_consensus.cc:2781] T 1487a2cb66e64f338dbfb7bbccf3eda3 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:45:54.566102 12914 raft_consensus.cc:482] T 1487a2cb66e64f338dbfb7bbccf3eda3 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:45:54.566165 12914 raft_consensus.cc:3037] T 1487a2cb66e64f338dbfb7bbccf3eda3 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:54.567329 12914 raft_consensus.cc:504] T 1487a2cb66e64f338dbfb7bbccf3eda3 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cbd8619d9048b5b56e4d9ea417288e" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 39727 } } peers { permanent_uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33151 } } peers { permanent_uuid: "fff12eae637c41ebabb8689441a6dbe5" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 34391 } }
I20260806 02:45:54.567765 12914 leader_election.cc:288] T 1487a2cb66e64f338dbfb7bbccf3eda3 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [CANDIDATE]: Term 1 election: Requested vote from peers e2cbd8619d9048b5b56e4d9ea417288e (127.1.183.131:39727), fff12eae637c41ebabb8689441a6dbe5 (127.1.183.129:34391)
I20260806 02:45:54.568209 12281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1487a2cb66e64f338dbfb7bbccf3eda3" candidate_uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fff12eae637c41ebabb8689441a6dbe5"
I20260806 02:45:54.568202 12688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1487a2cb66e64f338dbfb7bbccf3eda3" candidate_uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2cbd8619d9048b5b56e4d9ea417288e"
I20260806 02:45:54.568353 12688 raft_consensus.cc:3037] T 1487a2cb66e64f338dbfb7bbccf3eda3 P e2cbd8619d9048b5b56e4d9ea417288e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:54.568353 12281 raft_consensus.cc:3037] T 1487a2cb66e64f338dbfb7bbccf3eda3 P fff12eae637c41ebabb8689441a6dbe5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:54.569633 12281 raft_consensus.cc:2454] T 1487a2cb66e64f338dbfb7bbccf3eda3 P fff12eae637c41ebabb8689441a6dbe5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8dc07e3a1f4de49b0e6a5bec6878f2 in term 1.
I20260806 02:45:54.569633 12688 raft_consensus.cc:2454] T 1487a2cb66e64f338dbfb7bbccf3eda3 P e2cbd8619d9048b5b56e4d9ea417288e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8dc07e3a1f4de49b0e6a5bec6878f2 in term 1.
I20260806 02:45:54.569948 12367 leader_election.cc:302] T 1487a2cb66e64f338dbfb7bbccf3eda3 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8dc07e3a1f4de49b0e6a5bec6878f2, e2cbd8619d9048b5b56e4d9ea417288e; no voters:
I20260806 02:45:54.570134 12914 raft_consensus.cc:2781] T 1487a2cb66e64f338dbfb7bbccf3eda3 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:54.570427 12914 raft_consensus.cc:686] T 1487a2cb66e64f338dbfb7bbccf3eda3 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [term 1 LEADER]: Becoming Leader. State: Replica: 5e8dc07e3a1f4de49b0e6a5bec6878f2, State: Running, Role: LEADER
I20260806 02:45:54.570636 12914 consensus_queue.cc:237] T 1487a2cb66e64f338dbfb7bbccf3eda3 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cbd8619d9048b5b56e4d9ea417288e" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 39727 } } peers { permanent_uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33151 } } peers { permanent_uuid: "fff12eae637c41ebabb8689441a6dbe5" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 34391 } }
I20260806 02:45:54.571259 12872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:54.570983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.1.183.130:54301)
0806 02:45:54.571105 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:54.571176 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260806 02:45:54.573294 11828 catalog_manager.cc:5495] T 1487a2cb66e64f338dbfb7bbccf3eda3 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e8dc07e3a1f4de49b0e6a5bec6878f2 (127.1.183.130). New cstate: current_term: 1 leader_uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cbd8619d9048b5b56e4d9ea417288e" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 39727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8dc07e3a1f4de49b0e6a5bec6878f2" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fff12eae637c41ebabb8689441a6dbe5" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 34391 } health_report { overall_health: UNKNOWN } } }
I20260806 02:45:54.601889 12902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:54.598661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56506 (local address 127.1.183.130:33151)
0806 02:45:54.598804 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:54.598809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:54.598844 (+ 35us) server_negotiation.cc:407] Connection header received
0806 02:45:54.598915 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:54.598918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:54.598973 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:54.599057 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:54.599326 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:54.599858 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:54.600759 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:54.600994 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:54.601114 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:54.601385 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:54.601460 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:54.601544 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:54.601669 (+ 125us) server_negotiation.cc:299] Negotiation successful
0806 02:45:54.601728 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 02:45:54.604168 12281 raft_consensus.cc:1261] T 1487a2cb66e64f338dbfb7bbccf3eda3 P fff12eae637c41ebabb8689441a6dbe5 [term 1 FOLLOWER]: Refusing update from remote peer 5e8dc07e3a1f4de49b0e6a5bec6878f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:54.604135 12688 raft_consensus.cc:1261] T 1487a2cb66e64f338dbfb7bbccf3eda3 P e2cbd8619d9048b5b56e4d9ea417288e [term 1 FOLLOWER]: Refusing update from remote peer 5e8dc07e3a1f4de49b0e6a5bec6878f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:54.604658 12915 consensus_queue.cc:1035] T 1487a2cb66e64f338dbfb7bbccf3eda3 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fff12eae637c41ebabb8689441a6dbe5" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 34391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:54.604892 12914 consensus_queue.cc:1035] T 1487a2cb66e64f338dbfb7bbccf3eda3 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cbd8619d9048b5b56e4d9ea417288e" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 39727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:54.606546 12873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:54.606247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.1.183.129:41247)
0806 02:45:54.606382 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:54.606464 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 02:45:54.606884 12757 heartbeater.cc:503] Master 127.1.183.189:40231 was elected leader, sending a full tablet report...
I20260806 02:45:54.609098 12876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:54.608602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.1.183.131:53921)
0806 02:45:54.608944 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:54.609010 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252}
I20260806 02:45:54.609143 12872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:54.608635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.1.183.130:58993)
0806 02:45:54.609004 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:54.609071 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291}
I20260806 02:45:54.609472 12873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:54.609158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.1.183.129:46753)
0806 02:45:54.609330 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:54.609395 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260806 02:45:54.612198 12872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:54.611822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.1.183.130:51235)
0806 02:45:54.612040 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:54.612095 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260806 02:45:54.614142 11828 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:54.614329 11828 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:54.615170 12876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:54.614809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:39091 (local address 127.1.183.131:54691)
0806 02:45:54.615039 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:54.615096 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:39091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260806 02:45:54.619236 11828 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb3e8a85dcf24fb7b146b28dd2053249: Sending DeleteTablet for 3 replicas of tablet 1487a2cb66e64f338dbfb7bbccf3eda3
I20260806 02:45:54.620111 12439 tablet_service.cc:1514] Processing DeleteTablet for tablet 1487a2cb66e64f338dbfb7bbccf3eda3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:54 UTC) from {username='oryx', principal='oryx/127.1.183.189@KRBTEST.COM'} at 127.0.0.1:56494
I20260806 02:45:54.620213 12657 tablet_service.cc:1514] Processing DeleteTablet for tablet 1487a2cb66e64f338dbfb7bbccf3eda3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:54 UTC) from {username='oryx', principal='oryx/127.1.183.189@KRBTEST.COM'} at 127.0.0.1:42008
I20260806 02:45:54.620244 12438 tablet_service.cc:1514] Processing DeleteTablet for tablet 1487a2cb66e64f338dbfb7bbccf3eda3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:54 UTC) from {username='oryx', principal='oryx/127.1.183.189@KRBTEST.COM'} at 127.0.0.1:56494
I20260806 02:45:54.620441 12656 tablet_service.cc:1514] Processing DeleteTablet for tablet 1487a2cb66e64f338dbfb7bbccf3eda3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:54 UTC) from {username='oryx', principal='oryx/127.1.183.189@KRBTEST.COM'} at 127.0.0.1:42008
I20260806 02:45:54.620530 12935 tablet_replica.cc:330] T 1487a2cb66e64f338dbfb7bbccf3eda3 P 5e8dc07e3a1f4de49b0e6a5bec6878f2: stopping tablet replica
I20260806 02:45:54.620622 12936 tablet_replica.cc:330] T 1487a2cb66e64f338dbfb7bbccf3eda3 P e2cbd8619d9048b5b56e4d9ea417288e: stopping tablet replica
I20260806 02:45:54.620656 12935 raft_consensus.cc:2229] T 1487a2cb66e64f338dbfb7bbccf3eda3 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:45:54.620780 12936 raft_consensus.cc:2229] T 1487a2cb66e64f338dbfb7bbccf3eda3 P e2cbd8619d9048b5b56e4d9ea417288e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:54.620863 12935 raft_consensus.cc:2258] T 1487a2cb66e64f338dbfb7bbccf3eda3 P 5e8dc07e3a1f4de49b0e6a5bec6878f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:54.620918 12936 raft_consensus.cc:2258] T 1487a2cb66e64f338dbfb7bbccf3eda3 P e2cbd8619d9048b5b56e4d9ea417288e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:54.620965 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 12030
I20260806 02:45:54.621625 12935 ts_tablet_manager.cc:1905] T 1487a2cb66e64f338dbfb7bbccf3eda3 P 5e8dc07e3a1f4de49b0e6a5bec6878f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:54.621940 12936 ts_tablet_manager.cc:1905] T 1487a2cb66e64f338dbfb7bbccf3eda3 P e2cbd8619d9048b5b56e4d9ea417288e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:54.624758 12935 ts_tablet_manager.cc:1918] T 1487a2cb66e64f338dbfb7bbccf3eda3 P 5e8dc07e3a1f4de49b0e6a5bec6878f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:45:54.624847 12935 log.cc:1196] T 1487a2cb66e64f338dbfb7bbccf3eda3 P 5e8dc07e3a1f4de49b0e6a5bec6878f2: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-1/wal/wals/1487a2cb66e64f338dbfb7bbccf3eda3
I20260806 02:45:54.625128 12935 ts_tablet_manager.cc:1939] T 1487a2cb66e64f338dbfb7bbccf3eda3 P 5e8dc07e3a1f4de49b0e6a5bec6878f2: Deleting consensus metadata
I20260806 02:45:54.625540 12936 ts_tablet_manager.cc:1918] T 1487a2cb66e64f338dbfb7bbccf3eda3 P e2cbd8619d9048b5b56e4d9ea417288e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:45:54.625631 12936 log.cc:1196] T 1487a2cb66e64f338dbfb7bbccf3eda3 P e2cbd8619d9048b5b56e4d9ea417288e: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984313887056-1758-0/minicluster-data/ts-2/wal/wals/1487a2cb66e64f338dbfb7bbccf3eda3
I20260806 02:45:54.625972 12936 ts_tablet_manager.cc:1939] T 1487a2cb66e64f338dbfb7bbccf3eda3 P e2cbd8619d9048b5b56e4d9ea417288e: Deleting consensus metadata
I20260806 02:45:54.626926 11807 catalog_manager.cc:4841] TS e2cbd8619d9048b5b56e4d9ea417288e (127.1.183.131:39727): tablet 1487a2cb66e64f338dbfb7bbccf3eda3 (table test-table [id=ee6846dbebf64bb496e13bebb5e43869]) successfully deleted
W20260806 02:45:54.627247 11807 catalog_manager.cc:4808] TS e2cbd8619d9048b5b56e4d9ea417288e (127.1.183.131:39727): delete failed for tablet 1487a2cb66e64f338dbfb7bbccf3eda3 because the tablet was not found. No further retry: Not found: Tablet not found: 1487a2cb66e64f338dbfb7bbccf3eda3
I20260806 02:45:54.633332 11808 catalog_manager.cc:4841] TS 5e8dc07e3a1f4de49b0e6a5bec6878f2 (127.1.183.130:33151): tablet 1487a2cb66e64f338dbfb7bbccf3eda3 (table test-table [id=ee6846dbebf64bb496e13bebb5e43869]) successfully deleted
W20260806 02:45:54.633646 11808 catalog_manager.cc:4808] TS 5e8dc07e3a1f4de49b0e6a5bec6878f2 (127.1.183.130:33151): delete failed for tablet 1487a2cb66e64f338dbfb7bbccf3eda3 because the tablet was not found. No further retry: Not found: Tablet not found: 1487a2cb66e64f338dbfb7bbccf3eda3
W20260806 02:45:54.639210 11809 connection.cc:535] client connection to 127.1.183.129:34391 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 02:45:54.639593 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 12344
I20260806 02:45:54.642256 12901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:54.639824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:34391 (local address 127.0.0.1:49946)
0806 02:45:54.641992 (+ 2168us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:54.642109 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.129:34391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2044}
W20260806 02:45:54.645375 11809 catalog_manager.cc:4584] TS fff12eae637c41ebabb8689441a6dbe5 (127.1.183.129:34391): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1487a2cb66e64f338dbfb7bbccf3eda3: Network error: Client connection negotiation failed: client connection to 127.1.183.129:34391: connect: Connection refused (error 111)
I20260806 02:45:54.657697 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 12563
I20260806 02:45:54.669606 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 11634
I20260806 02:45:54.682677 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 11777
2026-08-06T02:45:54Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7694 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:45:54 dist-test-slave-gmjp krb5kdc[12940](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:45:54 dist-test-slave-gmjp krb5kdc[12940](info): set up 2 sockets
Aug 06 02:45:54 dist-test-slave-gmjp krb5kdc[12940](info): commencing operation
krb5kdc: starting...
W20260806 02:45:55.926158 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.210s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:45:56 dist-test-slave-gmjp krb5kdc[12940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:45:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:56Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.190.keytab.
Entry for principal oryx/127.1.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.190.keytab.
I20260806 02:45:56.067220 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:43847
--webserver_interface=127.1.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.190.keytab
--principal=oryx/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:40007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:43847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:56.187813 12958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:56.188164 12958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:56.188269 12958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:56.192399 12958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:56.192505 12958 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 02:45:56.192538 12958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:56.192576 12958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:56.192631 12958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:56.192672 12958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:56.198231 12958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:40007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:43847
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.190.keytab
--principal=oryx/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:43847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:56.199570 12958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:56.200939 12958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:56.209738 12968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:56.209726 12963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:56.209726 12966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:56.210105 12958 server_base.cc:1034] running on GCE node
I20260806 02:45:56.210645 12958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:56.211802 12958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:56.213003 12958 hybrid_clock.cc:648] HybridClock initialized: now 1785984356212994 us; error 34 us; skew 500 ppm
Aug 06 02:45:56 dist-test-slave-gmjp krb5kdc[12940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984356, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:56.218811 12958 init.cc:377] Logged in from keytab as oryx/127.1.183.190@KRBTEST.COM (short username oryx)
I20260806 02:45:56.219931 12958 webserver.cc:460] Webserver started at http://127.1.183.190:34843/ using document root <none> and password file <none>
I20260806 02:45:56.220633 12958 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:56.220749 12958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:56.220999 12958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:56.223249 12958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "f1557f14f7b246a3aee81db7b9d5a7eb"
format_stamp: "Formatted at 2026-08-06 02:45:56 on dist-test-slave-gmjp"
server_key: "e478e0fdadca67b3c2fb4c2a78d4be40"
server_key_iv: "fa4d21eb20aab1fed3bd39938e07371e"
server_key_version: "encryptionkey@0"
I20260806 02:45:56.223892 12958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "f1557f14f7b246a3aee81db7b9d5a7eb"
format_stamp: "Formatted at 2026-08-06 02:45:56 on dist-test-slave-gmjp"
server_key: "e478e0fdadca67b3c2fb4c2a78d4be40"
server_key_iv: "fa4d21eb20aab1fed3bd39938e07371e"
server_key_version: "encryptionkey@0"
I20260806 02:45:56.227999 12958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:45:56.230767 12976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:56.232333 12958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:56.232486 12958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "f1557f14f7b246a3aee81db7b9d5a7eb"
format_stamp: "Formatted at 2026-08-06 02:45:56 on dist-test-slave-gmjp"
server_key: "e478e0fdadca67b3c2fb4c2a78d4be40"
server_key_iv: "fa4d21eb20aab1fed3bd39938e07371e"
server_key_version: "encryptionkey@0"
I20260806 02:45:56.232609 12958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:56.262537 12958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:56.266497 12958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:56.266742 12958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:56.275626 12958 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:43847
I20260806 02:45:56.275698 13063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:43847 every 8 connection(s)
I20260806 02:45:56.276741 12958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:45:56.279803 13064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:56.285748 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 12958
I20260806 02:45:56.285915 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:45:56.285902 13064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1557f14f7b246a3aee81db7b9d5a7eb: Bootstrap starting.
I20260806 02:45:56.286238 1758 external_mini_cluster.cc:1428] Setting key ce52cad787e04d99e8d1660052fe946a
I20260806 02:45:56.288830 13064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1557f14f7b246a3aee81db7b9d5a7eb: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:56.289624 13064 log.cc:824] T 00000000000000000000000000000000 P f1557f14f7b246a3aee81db7b9d5a7eb: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:56.291828 13064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1557f14f7b246a3aee81db7b9d5a7eb: No bootstrap required, opened a new log
Aug 06 02:45:56 dist-test-slave-gmjp krb5kdc[12940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM
I20260806 02:45:56.296810 13064 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1557f14f7b246a3aee81db7b9d5a7eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1557f14f7b246a3aee81db7b9d5a7eb" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 43847 } }
I20260806 02:45:56.297122 13064 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1557f14f7b246a3aee81db7b9d5a7eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:56.297187 13064 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1557f14f7b246a3aee81db7b9d5a7eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1557f14f7b246a3aee81db7b9d5a7eb, State: Initialized, Role: FOLLOWER
I20260806 02:45:56.297760 13064 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1557f14f7b246a3aee81db7b9d5a7eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1557f14f7b246a3aee81db7b9d5a7eb" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 43847 } }
I20260806 02:45:56.297976 13064 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1557f14f7b246a3aee81db7b9d5a7eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:45:56.298072 13064 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1557f14f7b246a3aee81db7b9d5a7eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:45:56.298182 13064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1557f14f7b246a3aee81db7b9d5a7eb [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:56.299480 13064 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1557f14f7b246a3aee81db7b9d5a7eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1557f14f7b246a3aee81db7b9d5a7eb" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 43847 } }
I20260806 02:45:56.299889 13064 leader_election.cc:302] T 00000000000000000000000000000000 P f1557f14f7b246a3aee81db7b9d5a7eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1557f14f7b246a3aee81db7b9d5a7eb; no voters:
I20260806 02:45:56.300347 13064 leader_election.cc:288] T 00000000000000000000000000000000 P f1557f14f7b246a3aee81db7b9d5a7eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:45:56.300436 13072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1557f14f7b246a3aee81db7b9d5a7eb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:56.300658 13072 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1557f14f7b246a3aee81db7b9d5a7eb [term 1 LEADER]: Becoming Leader. State: Replica: f1557f14f7b246a3aee81db7b9d5a7eb, State: Running, Role: LEADER
I20260806 02:45:56.301026 13072 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1557f14f7b246a3aee81db7b9d5a7eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1557f14f7b246a3aee81db7b9d5a7eb" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 43847 } }
I20260806 02:45:56.301795 13064 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1557f14f7b246a3aee81db7b9d5a7eb [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:45:56.302331 13072 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1557f14f7b246a3aee81db7b9d5a7eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1557f14f7b246a3aee81db7b9d5a7eb. Latest consensus state: current_term: 1 leader_uuid: "f1557f14f7b246a3aee81db7b9d5a7eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1557f14f7b246a3aee81db7b9d5a7eb" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 43847 } } }
I20260806 02:45:56.302459 13072 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1557f14f7b246a3aee81db7b9d5a7eb [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:56.302915 13075 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1557f14f7b246a3aee81db7b9d5a7eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1557f14f7b246a3aee81db7b9d5a7eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1557f14f7b246a3aee81db7b9d5a7eb" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 43847 } } }
I20260806 02:45:56.303017 13075 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1557f14f7b246a3aee81db7b9d5a7eb [sys.catalog]: This master's current role is: LEADER
I20260806 02:45:56.303272 13069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:56.287708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48622 (local address 127.1.183.190:43847)
0806 02:45:56.288226 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:56.288244 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:56.288285 (+ 41us) server_negotiation.cc:407] Connection header received
0806 02:45:56.289050 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:56.289071 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:56.289431 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:56.289722 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:56.290650 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:56.291642 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:56.292882 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:56.293366 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:56.293488 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:56.296790 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:56.296796 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:56.296811 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:56.296842 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:56.299171 (+ 2329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:56.299667 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:56.299673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:56.299679 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:56.299828 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:56.300144 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:56.300173 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:56.300177 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:56.300771 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:56.301615 (+ 844us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:56.301742 (+ 127us) server_negotiation.cc:299] Negotiation successful
0806 02:45:56.302092 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":152,"threads_started":1}
W20260806 02:45:56.307032 13095 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f1557f14f7b246a3aee81db7b9d5a7eb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:45:56.307147 13095 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:45:56.307245 13079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:45:56.308210 13079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:45:56.314776 13079 catalog_manager.cc:1347] Generated new cluster ID: 2d2ad23d11ab46c5a8a1daa26c356c6d
I20260806 02:45:56.314870 13079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:45:56.343958 13079 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:45:56.345382 13079 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:45:56.356133 13079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1557f14f7b246a3aee81db7b9d5a7eb: Generated new TSK 0
I20260806 02:45:56.357156 13079 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.129.keytab.
Entry for principal oryx/127.1.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.129.keytab.
I20260806 02:45:56.399513 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.129:0
--local_ip_for_outbound_sockets=127.1.183.129
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:43847
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.129.keytab
--principal=oryx/127.1.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:40007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:56.521039 13108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:56.521313 13108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:56.521394 13108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:56.525533 13108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:56.525604 13108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:56.525645 13108 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:45:56.525712 13108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.129
I20260806 02:45:56.531085 13108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:40007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.129.keytab
--principal=oryx/127.1.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.183.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.183.190:43847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.129
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:56.532265 13108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:56.533396 13108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:56.543826 13115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:56.543843 13119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:56.543843 13117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:56.544114 13108 server_base.cc:1034] running on GCE node
I20260806 02:45:56.544692 13108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:56.545450 13108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:56.546648 13108 hybrid_clock.cc:648] HybridClock initialized: now 1785984356546634 us; error 40 us; skew 500 ppm
Aug 06 02:45:56 dist-test-slave-gmjp krb5kdc[12940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984356, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:56.549715 13108 init.cc:377] Logged in from keytab as oryx/127.1.183.129@KRBTEST.COM (short username oryx)
I20260806 02:45:56.550712 13108 webserver.cc:460] Webserver started at http://127.1.183.129:42509/ using document root <none> and password file <none>
I20260806 02:45:56.551352 13108 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:56.551458 13108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:56.551707 13108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:56.553987 13108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-0/data/instance:
uuid: "7d99f1b802814487b0455463222ed797"
format_stamp: "Formatted at 2026-08-06 02:45:56 on dist-test-slave-gmjp"
server_key: "7bcd50b87b02414775cb167394f21972"
server_key_iv: "738826626ce2b9592ecca0a483b08f6b"
server_key_version: "encryptionkey@0"
I20260806 02:45:56.554585 13108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance:
uuid: "7d99f1b802814487b0455463222ed797"
format_stamp: "Formatted at 2026-08-06 02:45:56 on dist-test-slave-gmjp"
server_key: "7bcd50b87b02414775cb167394f21972"
server_key_iv: "738826626ce2b9592ecca0a483b08f6b"
server_key_version: "encryptionkey@0"
I20260806 02:45:56.558555 13108 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260806 02:45:56.561187 13133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:56.562543 13108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:56.562700 13108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-0/wal
uuid: "7d99f1b802814487b0455463222ed797"
format_stamp: "Formatted at 2026-08-06 02:45:56 on dist-test-slave-gmjp"
server_key: "7bcd50b87b02414775cb167394f21972"
server_key_iv: "738826626ce2b9592ecca0a483b08f6b"
server_key_version: "encryptionkey@0"
I20260806 02:45:56.562827 13108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:56.579360 13108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:56.582692 13108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:56.582926 13108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:56.584595 13108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:56.584674 13108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:56.584748 13108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:56.584791 13108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:45:56 dist-test-slave-gmjp krb5kdc[12940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984356, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM
I20260806 02:45:56.597050 13108 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.129:43601
I20260806 02:45:56.597371 13303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.129:43601 every 8 connection(s)
I20260806 02:45:56.598197 13108 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
I20260806 02:45:56.600090 13069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:56.585682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:44543 (local address 127.1.183.190:43847)
0806 02:45:56.585863 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:56.585867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:56.587055 (+ 1188us) server_negotiation.cc:407] Connection header received
0806 02:45:56.588356 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:56.588360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:56.588432 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:56.588535 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:56.590171 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:56.590802 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:56.592304 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:56.592595 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:56.592743 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:56.595963 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:56.595967 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:56.595969 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:56.595997 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:56.597830 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:56.598424 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:56.598430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:56.598436 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:56.598534 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:56.598932 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:56.598936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:56.598938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:56.599191 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:56.599312 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:56.599842 (+ 530us) server_negotiation.cc:299] Negotiation successful
0806 02:45:56.599962 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260806 02:45:56.600900 13146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:56.586292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:43847 (local address 127.1.183.129:44543)
0806 02:45:56.586876 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:56.586925 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:56.588010 (+ 1085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:56.588815 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:56.588824 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:56.589309 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:56.589998 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:56.590015 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:56.590955 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:56.590960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:56.592124 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:56.592136 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:56.592809 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:56.592813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:56.593062 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:56.593843 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:56.593866 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:56.595750 (+ 1884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:56.597995 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:56.598001 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:56.598016 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:56.598317 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:56.598637 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:56.598640 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:56.598642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:56.598829 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:56.599316 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:56.599322 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:56.599612 (+ 290us) client_negotiation.cc:769] Sending connection context
0806 02:45:56.599774 (+ 162us) client_negotiation.cc:240] Negotiation successful
0806 02:45:56.600040 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":141,"threads_started":1}
I20260806 02:45:56.602352 13304 heartbeater.cc:347] Connected to a master server at 127.1.183.190:43847
I20260806 02:45:56.602555 13304 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:56.603163 13304 heartbeater.cc:511] Master 127.1.183.190:43847 requested a full tablet report, sending...
I20260806 02:45:56.604957 13000 ts_manager.cc:194] Registered new tserver with Master: 7d99f1b802814487b0455463222ed797 (127.1.183.129:43601)
I20260806 02:45:56.606526 13000 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.183.129@KRBTEST.COM'} at 127.1.183.129:44543
I20260806 02:45:56.608369 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 13108
I20260806 02:45:56.608525 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance
I20260806 02:45:56.608810 1758 external_mini_cluster.cc:1428] Setting key 51e77a9251286b6d5fe13c59bed83358
I20260806 02:45:56.616089 13069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:56.604869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:42583 (local address 127.1.183.190:43847)
0806 02:45:56.605016 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:56.605020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:56.605186 (+ 166us) server_negotiation.cc:407] Connection header received
0806 02:45:56.605282 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:56.605285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:56.605327 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:56.605410 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:56.605824 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:56.606603 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:56.607714 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:56.607945 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:56.608055 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:56.612534 (+ 4479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:56.612540 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:56.612544 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:56.612578 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:56.614200 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:56.614758 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:56.614760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:56.614761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:56.614835 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:56.615149 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:56.615151 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:56.615153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:56.615323 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:56.615413 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:56.615863 (+ 450us) server_negotiation.cc:299] Negotiation successful
0806 02:45:56.615967 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":72}
I20260806 02:45:56.616201 13310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:56.604799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:43847 (local address 127.1.183.129:42583)
0806 02:45:56.605113 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:56.605134 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:56.605217 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:56.605435 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:56.605437 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:56.605611 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:56.605712 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:56.605717 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:56.606761 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:56.606765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:56.607591 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:56.607601 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:56.608059 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:56.608062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:56.609279 (+ 1217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:56.609802 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:56.609816 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:56.610305 (+ 489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:56.614320 (+ 4015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:56.614327 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:56.614332 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:56.614656 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:56.614936 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:56.614942 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:56.614947 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:56.615056 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:56.615452 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:56.615458 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:56.615587 (+ 129us) client_negotiation.cc:769] Sending connection context
0806 02:45:56.615714 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 02:45:56.615895 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for oryx/127.1.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.130.keytab.
Entry for principal oryx/127.1.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.130.keytab.
I20260806 02:45:56.639987 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.130:0
--local_ip_for_outbound_sockets=127.1.183.130
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:43847
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.130.keytab
--principal=oryx/127.1.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:40007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:56.758472 13315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:56.758752 13315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:56.758800 13315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:56.762795 13315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:56.762861 13315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:56.762902 13315 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:45:56.762938 13315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.130
I20260806 02:45:56.768514 13315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:40007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.130.keytab
--principal=oryx/127.1.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.183.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.183.190:43847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.130
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:56.769767 13315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:56.770788 13315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:56.781082 13324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:56.781085 13325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:56.781104 13327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:56.781548 13315 server_base.cc:1034] running on GCE node
I20260806 02:45:56.782111 13315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:56.782825 13315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:56.784014 13315 hybrid_clock.cc:648] HybridClock initialized: now 1785984356783997 us; error 29 us; skew 500 ppm
Aug 06 02:45:56 dist-test-slave-gmjp krb5kdc[12940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984356, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:56.787051 13315 init.cc:377] Logged in from keytab as oryx/127.1.183.130@KRBTEST.COM (short username oryx)
I20260806 02:45:56.788105 13315 webserver.cc:460] Webserver started at http://127.1.183.130:46279/ using document root <none> and password file <none>
I20260806 02:45:56.788757 13315 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:56.788856 13315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:56.789119 13315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:56.791435 13315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-1/data/instance:
uuid: "0be05994eb164242b621cb037063e2d0"
format_stamp: "Formatted at 2026-08-06 02:45:56 on dist-test-slave-gmjp"
server_key: "1833c8905f642809137a0419307d0a62"
server_key_iv: "593c87249fb576c6704902898ea56aa7"
server_key_version: "encryptionkey@0"
I20260806 02:45:56.792100 13315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance:
uuid: "0be05994eb164242b621cb037063e2d0"
format_stamp: "Formatted at 2026-08-06 02:45:56 on dist-test-slave-gmjp"
server_key: "1833c8905f642809137a0419307d0a62"
server_key_iv: "593c87249fb576c6704902898ea56aa7"
server_key_version: "encryptionkey@0"
I20260806 02:45:56.796244 13315 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:45:56.799001 13338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:56.800438 13315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:56.800593 13315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-1/wal
uuid: "0be05994eb164242b621cb037063e2d0"
format_stamp: "Formatted at 2026-08-06 02:45:56 on dist-test-slave-gmjp"
server_key: "1833c8905f642809137a0419307d0a62"
server_key_iv: "593c87249fb576c6704902898ea56aa7"
server_key_version: "encryptionkey@0"
I20260806 02:45:56.800720 13315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:56.821269 13315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:56.825013 13315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:56.825243 13315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:56.826886 13315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:56.826964 13315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:56.827040 13315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:56.827085 13315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:45:56 dist-test-slave-gmjp krb5kdc[12940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984356, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM
I20260806 02:45:56.839329 13315 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.130:44967
I20260806 02:45:56.839681 13493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.130:44967 every 8 connection(s)
I20260806 02:45:56.840612 13315 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
I20260806 02:45:56.842798 13069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:56.827746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:56511 (local address 127.1.183.190:43847)
0806 02:45:56.827936 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:56.827944 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:56.829087 (+ 1143us) server_negotiation.cc:407] Connection header received
0806 02:45:56.830211 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:56.830217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:56.830288 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:56.830398 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:56.832411 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:56.833097 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:56.834295 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:56.834548 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:56.834662 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:56.838142 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:56.838148 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:56.838154 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:56.838188 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:56.840288 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:56.840983 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:56.840988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:56.840991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:56.841104 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:56.841562 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:56.841566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:56.841568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:56.841854 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:56.841952 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:56.842513 (+ 561us) server_negotiation.cc:299] Negotiation successful
0806 02:45:56.842652 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:45:56.843627 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:56.828398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:43847 (local address 127.1.183.130:56511)
0806 02:45:56.828915 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:56.828962 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:56.829932 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:56.830919 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:56.830928 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:56.831459 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:56.832230 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:56.832257 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:56.833232 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:56.833235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:56.834152 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:56.834163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:56.834754 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:56.834761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:56.834954 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:56.835647 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:56.835674 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:56.837860 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:56.840422 (+ 2562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:56.840428 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:56.840441 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:56.840848 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:56.841229 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:56.841232 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:56.841234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:56.841439 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:56.841966 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:56.841972 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:56.842216 (+ 244us) client_negotiation.cc:769] Sending connection context
0806 02:45:56.842381 (+ 165us) client_negotiation.cc:240] Negotiation successful
0806 02:45:56.842663 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":147,"threads_started":1}
I20260806 02:45:56.845305 13494 heartbeater.cc:347] Connected to a master server at 127.1.183.190:43847
I20260806 02:45:56.845489 13494 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:56.846071 13494 heartbeater.cc:511] Master 127.1.183.190:43847 requested a full tablet report, sending...
I20260806 02:45:56.847322 13000 ts_manager.cc:194] Registered new tserver with Master: 0be05994eb164242b621cb037063e2d0 (127.1.183.130:44967)
I20260806 02:45:56.848202 13000 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.183.130@KRBTEST.COM'} at 127.1.183.130:56511
I20260806 02:45:56.848513 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 13315
I20260806 02:45:56.848644 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance
I20260806 02:45:56.849020 1758 external_mini_cluster.cc:1428] Setting key 3219e2ba754e022339502e331a572048
I20260806 02:45:56.856312 13069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:56.847840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:54687 (local address 127.1.183.190:43847)
0806 02:45:56.847982 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:56.847985 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:56.848238 (+ 253us) server_negotiation.cc:407] Connection header received
0806 02:45:56.848461 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:56.848465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:56.848521 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:56.848618 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:56.849363 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:56.850064 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:56.851029 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:56.851411 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:56.851543 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:56.852772 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:56.852777 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:56.852780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:56.852806 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:56.854284 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:56.854765 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:56.854768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:56.854770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:56.854862 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:56.855149 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:56.855153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:56.855155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:56.855526 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:56.855644 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:56.855994 (+ 350us) server_negotiation.cc:299] Negotiation successful
0806 02:45:56.856101 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 02:45:56.856443 13503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:56.847763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:43847 (local address 127.1.183.130:54687)
0806 02:45:56.848122 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:56.848136 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:56.848281 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:56.848659 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:56.848662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:56.848903 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:56.849028 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:56.849035 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:56.850257 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:56.850261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:56.850900 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:56.850910 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:56.851555 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:56.851558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:56.851639 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:56.852248 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:56.852261 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:56.852611 (+ 350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:56.854428 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:56.854433 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:56.854435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:56.854652 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:56.854959 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:56.854962 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:56.854964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:56.855053 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:56.855670 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:56.855676 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:56.855807 (+ 131us) client_negotiation.cc:769] Sending connection context
0806 02:45:56.855938 (+ 131us) client_negotiation.cc:240] Negotiation successful
0806 02:45:56.856115 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":119,"threads_started":1}
WARNING: no policy specified for oryx/127.1.183.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.183.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.131.keytab.
Entry for principal oryx/127.1.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.131.keytab.
I20260806 02:45:56.881304 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.131:0
--local_ip_for_outbound_sockets=127.1.183.131
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:43847
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.131.keytab
--principal=oryx/127.1.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:40007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:57.002844 13506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:57.003217 13506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:57.003322 13506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:57.008298 13506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:57.008425 13506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:57.008512 13506 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:45:57.008611 13506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.131
I20260806 02:45:57.015558 13506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:40007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.131.keytab
--principal=oryx/127.1.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.183.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.183.190:43847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.131
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:57.017009 13506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:57.018236 13506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:57.028818 13516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:57.028820 13511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:57.028820 13513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:57.029160 13506 server_base.cc:1034] running on GCE node
I20260806 02:45:57.029600 13506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:57.030319 13506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:57.031524 13506 hybrid_clock.cc:648] HybridClock initialized: now 1785984357031502 us; error 34 us; skew 500 ppm
Aug 06 02:45:57 dist-test-slave-gmjp krb5kdc[12940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984357, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:45:57.034631 13506 init.cc:377] Logged in from keytab as oryx/127.1.183.131@KRBTEST.COM (short username oryx)
I20260806 02:45:57.035702 13506 webserver.cc:460] Webserver started at http://127.1.183.131:46753/ using document root <none> and password file <none>
I20260806 02:45:57.036448 13506 fs_manager.cc:359] Metadata directory not provided
I20260806 02:45:57.036549 13506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:45:57.036827 13506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:45:57.039111 13506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-2/data/instance:
uuid: "64a56e7a8ca6492680dd530522f4a3f9"
format_stamp: "Formatted at 2026-08-06 02:45:57 on dist-test-slave-gmjp"
server_key: "4d07a4b97b6e2122dc3911e134819458"
server_key_iv: "976fef8e688df3f1e6566bc04a4f4fd6"
server_key_version: "encryptionkey@0"
I20260806 02:45:57.039752 13506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance:
uuid: "64a56e7a8ca6492680dd530522f4a3f9"
format_stamp: "Formatted at 2026-08-06 02:45:57 on dist-test-slave-gmjp"
server_key: "4d07a4b97b6e2122dc3911e134819458"
server_key_iv: "976fef8e688df3f1e6566bc04a4f4fd6"
server_key_version: "encryptionkey@0"
I20260806 02:45:57.044050 13506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:45:57.047024 13523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:57.048461 13506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:45:57.048611 13506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-2/wal
uuid: "64a56e7a8ca6492680dd530522f4a3f9"
format_stamp: "Formatted at 2026-08-06 02:45:57 on dist-test-slave-gmjp"
server_key: "4d07a4b97b6e2122dc3911e134819458"
server_key_iv: "976fef8e688df3f1e6566bc04a4f4fd6"
server_key_version: "encryptionkey@0"
I20260806 02:45:57.048745 13506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:45:57.067359 13506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:45:57.070686 13506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:45:57.070927 13506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:45:57.072695 13506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:45:57.072777 13506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:45:57.072857 13506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:45:57.072903 13506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:45:57 dist-test-slave-gmjp krb5kdc[12940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984357, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM
I20260806 02:45:57.084774 13506 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.131:33073
I20260806 02:45:57.084982 13693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.131:33073 every 8 connection(s)
I20260806 02:45:57.085844 13506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
I20260806 02:45:57.088451 13069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:57.073522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:55037 (local address 127.1.183.190:43847)
0806 02:45:57.073701 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:57.073706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:57.074602 (+ 896us) server_negotiation.cc:407] Connection header received
0806 02:45:57.075642 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:57.075649 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:57.075721 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:57.075841 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:57.077357 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.078111 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.079540 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.079777 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.079893 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.084090 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:57.084095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:57.084097 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:57.084135 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:57.086042 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:57.086728 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:57.086732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:57.086734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:57.086835 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:57.087254 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:57.087256 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:57.087258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:57.087471 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:57.087550 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:57.088094 (+ 544us) server_negotiation.cc:299] Negotiation successful
0806 02:45:57.088311 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 02:45:57.089417 13538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:57.073925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:43847 (local address 127.1.183.131:55037)
0806 02:45:57.074412 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:57.074466 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:57.075368 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:57.076148 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:57.076197 (+ 49us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:57.076596 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:57.077197 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:57.077213 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.078241 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.078245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:57.079405 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:57.079423 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.079944 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.079949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:57.080134 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.080914 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:57.080940 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:57.083881 (+ 2941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:57.086180 (+ 2299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:57.086187 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:57.086203 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:57.086589 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:57.086969 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:57.086972 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:57.086974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:57.087133 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:57.087579 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:57.087584 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:57.087862 (+ 278us) client_negotiation.cc:769] Sending connection context
0806 02:45:57.088047 (+ 185us) client_negotiation.cc:240] Negotiation successful
0806 02:45:57.088473 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":136,"threads_started":1}
I20260806 02:45:57.089479 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 13506
I20260806 02:45:57.089619 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance
I20260806 02:45:57.089929 1758 external_mini_cluster.cc:1428] Setting key 672d8e9351440b08f6133bcb1eabbe72
I20260806 02:45:57.091198 13695 heartbeater.cc:347] Connected to a master server at 127.1.183.190:43847
I20260806 02:45:57.091379 13695 heartbeater.cc:464] Registering TS with master...
I20260806 02:45:57.092094 13695 heartbeater.cc:511] Master 127.1.183.190:43847 requested a full tablet report, sending...
I20260806 02:45:57.093268 12997 ts_manager.cc:194] Registered new tserver with Master: 64a56e7a8ca6492680dd530522f4a3f9 (127.1.183.131:33073)
I20260806 02:45:57.094098 12997 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.183.131@KRBTEST.COM'} at 127.1.183.131:55037
I20260806 02:45:57.100870 13069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:57.093759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:54755 (local address 127.1.183.190:43847)
0806 02:45:57.093929 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:57.093934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:57.094089 (+ 155us) server_negotiation.cc:407] Connection header received
0806 02:45:57.094192 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:57.094195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:57.094234 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:57.094316 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:57.094849 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.095588 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.096562 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.096771 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.096851 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.097950 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:57.097953 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:57.097955 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:57.097977 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:57.099307 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:57.099785 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:57.099787 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:57.099788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:57.099861 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:57.100108 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:57.100110 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:57.100111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:57.100339 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:57.100427 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:57.100657 (+ 230us) server_negotiation.cc:299] Negotiation successful
0806 02:45:57.100754 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 02:45:57.100884 13706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:57.093684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:43847 (local address 127.1.183.131:54755)
0806 02:45:57.093998 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:57.094012 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:57.094106 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:57.094367 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:57.094374 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:57.094620 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:57.094730 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:57.094736 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.095721 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.095724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:57.096451 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:57.096461 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.096876 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.096878 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:57.096952 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.097448 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:57.097460 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:57.097821 (+ 361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:57.099440 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:57.099444 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:57.099446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:57.099684 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:57.099945 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:57.099947 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:57.099948 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:57.100030 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:57.100426 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:57.100428 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:57.100511 (+ 83us) client_negotiation.cc:769] Sending connection context
0806 02:45:57.100598 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 02:45:57.100738 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":105,"threads_started":1}
I20260806 02:45:57.104288 1758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:45:57 dist-test-slave-gmjp krb5kdc[12940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984357, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:45:57 dist-test-slave-gmjp krb5kdc[12940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:45:57 dist-test-slave-gmjp krb5kdc[12940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 02:45:57.131210 13069 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:57.124459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48624 (local address 127.1.183.190:43847)
0806 02:45:57.124617 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:57.124621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:57.124634 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:45:57.124678 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:57.124681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:57.124720 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:57.124820 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:57.125621 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.126161 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.127068 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.127308 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.127439 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.131034 (+ 3595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48624: BlockingRecv error: recv got EOF from 127.0.0.1:48624 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 06 02:45:57 dist-test-slave-gmjp krb5kdc[12940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984357, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM
I20260806 02:45:57.141197 13069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:57.132854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48636 (local address 127.1.183.190:43847)
0806 02:45:57.132986 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:57.132989 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:57.133003 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:45:57.133121 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:57.133123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:57.133163 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:57.133229 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:57.133821 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.134310 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.135084 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.135286 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.135361 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.138010 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:57.138014 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:57.138020 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:57.138050 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:57.139415 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:57.139828 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:57.139830 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:57.139832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:57.139908 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:57.140273 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:57.140275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:57.140276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:57.140480 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:57.140588 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:57.140916 (+ 328us) server_negotiation.cc:299] Negotiation successful
0806 02:45:57.141026 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:45:57.144692 12997 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48636:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:45:57.148082 12997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:45:57.164202 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:57.157879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37104 (local address 127.1.183.131:33073)
0806 02:45:57.158768 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:57.158781 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:57.158809 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:45:57.158943 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:57.158950 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:57.159182 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:57.159374 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:57.159693 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.160991 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.162844 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.163418 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.163555 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.163752 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:57.163869 (+ 117us) server_negotiation.cc:299] Negotiation successful
0806 02:45:57.163991 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":772,"thread_start_us":105,"threads_started":1}
I20260806 02:45:57.164180 13729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:57.157538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:44967 (local address 127.0.0.1:43500)
0806 02:45:57.158146 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:57.158202 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:57.158337 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:57.159436 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:57.159441 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:57.159463 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:57.159572 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:57.159578 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.161150 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.161155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:57.162563 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:57.162576 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.163699 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.163703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:57.163794 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.163817 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 02:45:57.163877 (+ 60us) client_negotiation.cc:240] Negotiation successful
0806 02:45:57.164000 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":174,"thread_start_us":104,"threads_started":1}
I20260806 02:45:57.164535 13734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:57.158581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35030 (local address 127.1.183.129:43601)
0806 02:45:57.159127 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:57.159134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:57.159734 (+ 600us) server_negotiation.cc:407] Connection header received
0806 02:45:57.159849 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:57.159860 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:57.159996 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:57.160142 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:57.160365 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.161096 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.162825 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.163556 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.163672 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.163838 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:57.164265 (+ 427us) server_negotiation.cc:299] Negotiation successful
0806 02:45:57.164397 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":98,"threads_started":1}
I20260806 02:45:57.164685 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:57.158299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43500 (local address 127.1.183.130:44967)
0806 02:45:57.158934 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:57.158943 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:57.158987 (+ 44us) server_negotiation.cc:407] Connection header received
0806 02:45:57.159066 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:57.159071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:57.159258 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:57.159813 (+ 555us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:57.159952 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.161021 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.162675 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.163548 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.163982 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.164200 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:57.164422 (+ 222us) server_negotiation.cc:299] Negotiation successful
0806 02:45:57.164518 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":154,"threads_started":1}
I20260806 02:45:57.164856 13728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:57.158366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:43601 (local address 127.0.0.1:35030)
0806 02:45:57.159642 (+ 1276us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:57.159657 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:57.159741 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:57.160127 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:57.160130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:57.160148 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:57.160256 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:57.160265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.161214 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.161217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:57.162681 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:57.162693 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.163904 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.163908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:57.163989 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.164009 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 02:45:57.164074 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 02:45:57.164179 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1201,"mutex_wait_us":56,"thread_start_us":111,"threads_started":1}
I20260806 02:45:57.165922 13733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:57.157723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:33073 (local address 127.0.0.1:37104)
0806 02:45:57.158613 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:57.158628 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:57.158713 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:57.159367 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:57.159373 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:57.159396 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:57.159549 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:57.159557 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.161140 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.161143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:57.162681 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:57.162693 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.163553 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.163556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:57.163668 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.163706 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 02:45:57.163836 (+ 130us) client_negotiation.cc:240] Negotiation successful
0806 02:45:57.165734 (+ 1898us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":123,"thread_start_us":124,"threads_started":1}
I20260806 02:45:57.167856 13404 tablet_service.cc:1467] Processing CreateTablet for tablet 13ed5317b3e54d89a8b3635ee75cfe81 (DEFAULT_TABLE table=test-table [id=e622254399c540c1b6f0ee48dbb2e1b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:57.168480 13599 tablet_service.cc:1467] Processing CreateTablet for tablet 13ed5317b3e54d89a8b3635ee75cfe81 (DEFAULT_TABLE table=test-table [id=e622254399c540c1b6f0ee48dbb2e1b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:57.169019 13404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13ed5317b3e54d89a8b3635ee75cfe81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:57.169562 13599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13ed5317b3e54d89a8b3635ee75cfe81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:57.170226 13201 tablet_service.cc:1467] Processing CreateTablet for tablet 13ed5317b3e54d89a8b3635ee75cfe81 (DEFAULT_TABLE table=test-table [id=e622254399c540c1b6f0ee48dbb2e1b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:57.171242 13201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13ed5317b3e54d89a8b3635ee75cfe81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:57.177186 13735 tablet_bootstrap.cc:492] T 13ed5317b3e54d89a8b3635ee75cfe81 P 0be05994eb164242b621cb037063e2d0: Bootstrap starting.
I20260806 02:45:57.177986 13737 tablet_bootstrap.cc:492] T 13ed5317b3e54d89a8b3635ee75cfe81 P 64a56e7a8ca6492680dd530522f4a3f9: Bootstrap starting.
I20260806 02:45:57.178977 13738 tablet_bootstrap.cc:492] T 13ed5317b3e54d89a8b3635ee75cfe81 P 7d99f1b802814487b0455463222ed797: Bootstrap starting.
I20260806 02:45:57.179921 13735 tablet_bootstrap.cc:654] T 13ed5317b3e54d89a8b3635ee75cfe81 P 0be05994eb164242b621cb037063e2d0: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:57.180090 13737 tablet_bootstrap.cc:654] T 13ed5317b3e54d89a8b3635ee75cfe81 P 64a56e7a8ca6492680dd530522f4a3f9: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:57.180902 13738 tablet_bootstrap.cc:654] T 13ed5317b3e54d89a8b3635ee75cfe81 P 7d99f1b802814487b0455463222ed797: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:57.180930 13735 log.cc:824] T 13ed5317b3e54d89a8b3635ee75cfe81 P 0be05994eb164242b621cb037063e2d0: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:57.180935 13737 log.cc:824] T 13ed5317b3e54d89a8b3635ee75cfe81 P 64a56e7a8ca6492680dd530522f4a3f9: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:57.181748 13738 log.cc:824] T 13ed5317b3e54d89a8b3635ee75cfe81 P 7d99f1b802814487b0455463222ed797: Log is configured to *not* fsync() on all Append() calls
I20260806 02:45:57.183472 13735 tablet_bootstrap.cc:492] T 13ed5317b3e54d89a8b3635ee75cfe81 P 0be05994eb164242b621cb037063e2d0: No bootstrap required, opened a new log
I20260806 02:45:57.183696 13735 ts_tablet_manager.cc:1397] T 13ed5317b3e54d89a8b3635ee75cfe81 P 0be05994eb164242b621cb037063e2d0: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260806 02:45:57.184115 13737 tablet_bootstrap.cc:492] T 13ed5317b3e54d89a8b3635ee75cfe81 P 64a56e7a8ca6492680dd530522f4a3f9: No bootstrap required, opened a new log
I20260806 02:45:57.184355 13737 ts_tablet_manager.cc:1397] T 13ed5317b3e54d89a8b3635ee75cfe81 P 64a56e7a8ca6492680dd530522f4a3f9: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 02:45:57.188117 13735 raft_consensus.cc:348] T 13ed5317b3e54d89a8b3635ee75cfe81 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } } peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } }
I20260806 02:45:57.188453 13735 raft_consensus.cc:374] T 13ed5317b3e54d89a8b3635ee75cfe81 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:57.188524 13735 raft_consensus.cc:729] T 13ed5317b3e54d89a8b3635ee75cfe81 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0be05994eb164242b621cb037063e2d0, State: Initialized, Role: FOLLOWER
I20260806 02:45:57.189066 13735 consensus_queue.cc:260] T 13ed5317b3e54d89a8b3635ee75cfe81 P 0be05994eb164242b621cb037063e2d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } } peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } }
I20260806 02:45:57.188731 13737 raft_consensus.cc:348] T 13ed5317b3e54d89a8b3635ee75cfe81 P 64a56e7a8ca6492680dd530522f4a3f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } } peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } }
I20260806 02:45:57.189268 13737 raft_consensus.cc:374] T 13ed5317b3e54d89a8b3635ee75cfe81 P 64a56e7a8ca6492680dd530522f4a3f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:57.189327 13737 raft_consensus.cc:729] T 13ed5317b3e54d89a8b3635ee75cfe81 P 64a56e7a8ca6492680dd530522f4a3f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64a56e7a8ca6492680dd530522f4a3f9, State: Initialized, Role: FOLLOWER
I20260806 02:45:57.189783 13737 consensus_queue.cc:260] T 13ed5317b3e54d89a8b3635ee75cfe81 P 64a56e7a8ca6492680dd530522f4a3f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } } peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } }
I20260806 02:45:57.189903 13494 heartbeater.cc:503] Master 127.1.183.190:43847 was elected leader, sending a full tablet report...
I20260806 02:45:57.190296 13735 ts_tablet_manager.cc:1428] T 13ed5317b3e54d89a8b3635ee75cfe81 P 0be05994eb164242b621cb037063e2d0: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20260806 02:45:57.190871 13737 ts_tablet_manager.cc:1428] T 13ed5317b3e54d89a8b3635ee75cfe81 P 64a56e7a8ca6492680dd530522f4a3f9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:45:57.191349 13695 heartbeater.cc:503] Master 127.1.183.190:43847 was elected leader, sending a full tablet report...
I20260806 02:45:57.192553 13738 tablet_bootstrap.cc:492] T 13ed5317b3e54d89a8b3635ee75cfe81 P 7d99f1b802814487b0455463222ed797: No bootstrap required, opened a new log
I20260806 02:45:57.192835 13738 ts_tablet_manager.cc:1397] T 13ed5317b3e54d89a8b3635ee75cfe81 P 7d99f1b802814487b0455463222ed797: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20260806 02:45:57.197161 13738 raft_consensus.cc:348] T 13ed5317b3e54d89a8b3635ee75cfe81 P 7d99f1b802814487b0455463222ed797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } } peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } }
I20260806 02:45:57.197399 13738 raft_consensus.cc:374] T 13ed5317b3e54d89a8b3635ee75cfe81 P 7d99f1b802814487b0455463222ed797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:57.197443 13738 raft_consensus.cc:729] T 13ed5317b3e54d89a8b3635ee75cfe81 P 7d99f1b802814487b0455463222ed797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d99f1b802814487b0455463222ed797, State: Initialized, Role: FOLLOWER
I20260806 02:45:57.197866 13738 consensus_queue.cc:260] T 13ed5317b3e54d89a8b3635ee75cfe81 P 7d99f1b802814487b0455463222ed797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } } peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } }
I20260806 02:45:57.198652 13304 heartbeater.cc:503] Master 127.1.183.190:43847 was elected leader, sending a full tablet report...
I20260806 02:45:57.198979 13738 ts_tablet_manager.cc:1428] T 13ed5317b3e54d89a8b3635ee75cfe81 P 7d99f1b802814487b0455463222ed797: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:45:57.235518 13742 raft_consensus.cc:482] T 13ed5317b3e54d89a8b3635ee75cfe81 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:45:57.235780 13742 raft_consensus.cc:504] T 13ed5317b3e54d89a8b3635ee75cfe81 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } } peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } }
I20260806 02:45:57.237160 13742 leader_election.cc:288] T 13ed5317b3e54d89a8b3635ee75cfe81 P 0be05994eb164242b621cb037063e2d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64a56e7a8ca6492680dd530522f4a3f9 (127.1.183.131:33073), 7d99f1b802814487b0455463222ed797 (127.1.183.129:43601)
I20260806 02:45:57.241811 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:57.237417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:33073 (local address 127.1.183.130:50627)
0806 02:45:57.237543 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:57.237561 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:57.237662 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:57.238210 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:57.238212 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:57.238233 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:57.238405 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:57.238413 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.239358 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.239361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:57.240368 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:57.240378 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.241240 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.241243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:57.241404 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.241487 (+ 83us) client_negotiation.cc:769] Sending connection context
0806 02:45:57.241569 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 02:45:57.241652 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":35}
I20260806 02:45:57.241847 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:57.237616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:50627 (local address 127.1.183.131:33073)
0806 02:45:57.237826 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:57.237832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:57.237852 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:45:57.237912 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:57.237914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:57.237964 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:57.238054 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:57.238554 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.239231 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.240504 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.241111 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.241236 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.241304 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:57.241644 (+ 340us) server_negotiation.cc:299] Negotiation successful
0806 02:45:57.241740 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:45:57.242707 13734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:57.237959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:53337 (local address 127.1.183.129:43601)
0806 02:45:57.238087 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:57.238091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:57.238322 (+ 231us) server_negotiation.cc:407] Connection header received
0806 02:45:57.238426 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:57.238429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:57.238511 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:57.238591 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:57.238932 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.239533 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.241493 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.242127 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.242252 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.242320 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:57.242527 (+ 207us) server_negotiation.cc:299] Negotiation successful
0806 02:45:57.242602 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:45:57.242714 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:57.237695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:43601 (local address 127.1.183.130:53337)
0806 02:45:57.238191 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:57.238206 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:57.238328 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:57.238620 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:57.238622 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:57.238643 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:57.238768 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:57.238774 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.239649 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.239652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:57.241351 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:57.241368 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.242283 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.242287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:57.242391 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.242416 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 02:45:57.242501 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 02:45:57.242566 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":26,"spinlock_wait_cycles":3968,"thread_start_us":177,"threads_started":1}
I20260806 02:45:57.242674 13627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ed5317b3e54d89a8b3635ee75cfe81" candidate_uuid: "0be05994eb164242b621cb037063e2d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64a56e7a8ca6492680dd530522f4a3f9" is_pre_election: true
I20260806 02:45:57.243012 13627 raft_consensus.cc:2454] T 13ed5317b3e54d89a8b3635ee75cfe81 P 64a56e7a8ca6492680dd530522f4a3f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0be05994eb164242b621cb037063e2d0 in term 0.
I20260806 02:45:57.243525 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ed5317b3e54d89a8b3635ee75cfe81" candidate_uuid: "0be05994eb164242b621cb037063e2d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d99f1b802814487b0455463222ed797" is_pre_election: true
I20260806 02:45:57.243572 13344 leader_election.cc:302] T 13ed5317b3e54d89a8b3635ee75cfe81 P 0be05994eb164242b621cb037063e2d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0be05994eb164242b621cb037063e2d0, 64a56e7a8ca6492680dd530522f4a3f9; no voters:
I20260806 02:45:57.243840 13234 raft_consensus.cc:2454] T 13ed5317b3e54d89a8b3635ee75cfe81 P 7d99f1b802814487b0455463222ed797 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0be05994eb164242b621cb037063e2d0 in term 0.
I20260806 02:45:57.243875 13742 raft_consensus.cc:2781] T 13ed5317b3e54d89a8b3635ee75cfe81 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:45:57.243959 13742 raft_consensus.cc:482] T 13ed5317b3e54d89a8b3635ee75cfe81 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:45:57.244015 13742 raft_consensus.cc:3037] T 13ed5317b3e54d89a8b3635ee75cfe81 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:57.245314 13742 raft_consensus.cc:504] T 13ed5317b3e54d89a8b3635ee75cfe81 P 0be05994eb164242b621cb037063e2d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } } peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } }
I20260806 02:45:57.245661 13742 leader_election.cc:288] T 13ed5317b3e54d89a8b3635ee75cfe81 P 0be05994eb164242b621cb037063e2d0 [CANDIDATE]: Term 1 election: Requested vote from peers 64a56e7a8ca6492680dd530522f4a3f9 (127.1.183.131:33073), 7d99f1b802814487b0455463222ed797 (127.1.183.129:43601)
I20260806 02:45:57.246063 13627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ed5317b3e54d89a8b3635ee75cfe81" candidate_uuid: "0be05994eb164242b621cb037063e2d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64a56e7a8ca6492680dd530522f4a3f9"
I20260806 02:45:57.246205 13627 raft_consensus.cc:3037] T 13ed5317b3e54d89a8b3635ee75cfe81 P 64a56e7a8ca6492680dd530522f4a3f9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:57.246187 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ed5317b3e54d89a8b3635ee75cfe81" candidate_uuid: "0be05994eb164242b621cb037063e2d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d99f1b802814487b0455463222ed797"
I20260806 02:45:57.246345 13234 raft_consensus.cc:3037] T 13ed5317b3e54d89a8b3635ee75cfe81 P 7d99f1b802814487b0455463222ed797 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:57.247498 13234 raft_consensus.cc:2454] T 13ed5317b3e54d89a8b3635ee75cfe81 P 7d99f1b802814487b0455463222ed797 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0be05994eb164242b621cb037063e2d0 in term 1.
I20260806 02:45:57.247651 13627 raft_consensus.cc:2454] T 13ed5317b3e54d89a8b3635ee75cfe81 P 64a56e7a8ca6492680dd530522f4a3f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0be05994eb164242b621cb037063e2d0 in term 1.
I20260806 02:45:57.247856 13344 leader_election.cc:302] T 13ed5317b3e54d89a8b3635ee75cfe81 P 0be05994eb164242b621cb037063e2d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0be05994eb164242b621cb037063e2d0, 7d99f1b802814487b0455463222ed797; no voters:
I20260806 02:45:57.248061 13742 raft_consensus.cc:2781] T 13ed5317b3e54d89a8b3635ee75cfe81 P 0be05994eb164242b621cb037063e2d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:57.248343 13742 raft_consensus.cc:686] T 13ed5317b3e54d89a8b3635ee75cfe81 P 0be05994eb164242b621cb037063e2d0 [term 1 LEADER]: Becoming Leader. State: Replica: 0be05994eb164242b621cb037063e2d0, State: Running, Role: LEADER
I20260806 02:45:57.248698 13742 consensus_queue.cc:237] T 13ed5317b3e54d89a8b3635ee75cfe81 P 0be05994eb164242b621cb037063e2d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } } peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } }
I20260806 02:45:57.252271 13000 catalog_manager.cc:5495] T 13ed5317b3e54d89a8b3635ee75cfe81 P 0be05994eb164242b621cb037063e2d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0be05994eb164242b621cb037063e2d0 (127.1.183.130). New cstate: current_term: 1 leader_uuid: "0be05994eb164242b621cb037063e2d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } health_report { overall_health: UNKNOWN } } }
I20260806 02:45:57.265460 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:57.262113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43514 (local address 127.1.183.130:44967)
0806 02:45:57.262243 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:57.262246 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:57.262258 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:45:57.262343 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:57.262346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:57.262398 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:57.262490 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:57.262722 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.263230 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.264060 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.264348 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.264440 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.264670 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:57.265129 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:57.265220 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:57.265281 (+ 61us) server_negotiation.cc:299] Negotiation successful
0806 02:45:57.265344 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:45:57.272007 13234 raft_consensus.cc:1261] T 13ed5317b3e54d89a8b3635ee75cfe81 P 7d99f1b802814487b0455463222ed797 [term 1 FOLLOWER]: Refusing update from remote peer 0be05994eb164242b621cb037063e2d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:57.272055 13627 raft_consensus.cc:1261] T 13ed5317b3e54d89a8b3635ee75cfe81 P 64a56e7a8ca6492680dd530522f4a3f9 [term 1 FOLLOWER]: Refusing update from remote peer 0be05994eb164242b621cb037063e2d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:57.272848 13742 consensus_queue.cc:1035] T 13ed5317b3e54d89a8b3635ee75cfe81 P 0be05994eb164242b621cb037063e2d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:57.273162 13749 consensus_queue.cc:1035] T 13ed5317b3e54d89a8b3635ee75cfe81 P 0be05994eb164242b621cb037063e2d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:57.282763 13757 mvcc.cc:205] Tried to move back new op lower bound from 7315391927378096128 to 7315391927293554688. Current Snapshot: MvccSnapshot[applied={T|T < 7315391927378096128}]
I20260806 02:45:57.290731 13758 mvcc.cc:205] Tried to move back new op lower bound from 7315391927378096128 to 7315391927293554688. Current Snapshot: MvccSnapshot[applied={T|T < 7315391927378096128}]
I20260806 02:45:57.291383 12997 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:57.291561 12997 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:57.294359 12997 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1557f14f7b246a3aee81db7b9d5a7eb: Sending DeleteTablet for 3 replicas of tablet 13ed5317b3e54d89a8b3635ee75cfe81
I20260806 02:45:57.295562 13599 tablet_service.cc:1514] Processing DeleteTablet for tablet 13ed5317b3e54d89a8b3635ee75cfe81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:57 UTC) from {username='oryx', principal='oryx/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:37104
I20260806 02:45:57.295675 13404 tablet_service.cc:1514] Processing DeleteTablet for tablet 13ed5317b3e54d89a8b3635ee75cfe81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:57 UTC) from {username='oryx', principal='oryx/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:43500
I20260806 02:45:57.296133 13201 tablet_service.cc:1514] Processing DeleteTablet for tablet 13ed5317b3e54d89a8b3635ee75cfe81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:57 UTC) from {username='oryx', principal='oryx/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:35030
I20260806 02:45:57.296487 13774 tablet_replica.cc:330] T 13ed5317b3e54d89a8b3635ee75cfe81 P 0be05994eb164242b621cb037063e2d0: stopping tablet replica
I20260806 02:45:57.296736 13772 tablet_replica.cc:330] T 13ed5317b3e54d89a8b3635ee75cfe81 P 64a56e7a8ca6492680dd530522f4a3f9: stopping tablet replica
I20260806 02:45:57.297108 13772 raft_consensus.cc:2229] T 13ed5317b3e54d89a8b3635ee75cfe81 P 64a56e7a8ca6492680dd530522f4a3f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:57.297382 13774 raft_consensus.cc:2229] T 13ed5317b3e54d89a8b3635ee75cfe81 P 0be05994eb164242b621cb037063e2d0 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:45:57.297498 13772 raft_consensus.cc:2258] T 13ed5317b3e54d89a8b3635ee75cfe81 P 64a56e7a8ca6492680dd530522f4a3f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:57.297544 12997 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48636:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 02:45:57.297919 13774 raft_consensus.cc:2258] T 13ed5317b3e54d89a8b3635ee75cfe81 P 0be05994eb164242b621cb037063e2d0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260806 02:45:57.298035 12997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:45:57.299547 13774 ts_tablet_manager.cc:1905] T 13ed5317b3e54d89a8b3635ee75cfe81 P 0be05994eb164242b621cb037063e2d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:57.299548 13772 ts_tablet_manager.cc:1905] T 13ed5317b3e54d89a8b3635ee75cfe81 P 64a56e7a8ca6492680dd530522f4a3f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:57.302716 13774 ts_tablet_manager.cc:1918] T 13ed5317b3e54d89a8b3635ee75cfe81 P 0be05994eb164242b621cb037063e2d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:45:57.302842 13774 log.cc:1196] T 13ed5317b3e54d89a8b3635ee75cfe81 P 0be05994eb164242b621cb037063e2d0: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-1/wal/wals/13ed5317b3e54d89a8b3635ee75cfe81
I20260806 02:45:57.302925 13772 ts_tablet_manager.cc:1918] T 13ed5317b3e54d89a8b3635ee75cfe81 P 64a56e7a8ca6492680dd530522f4a3f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:45:57.303151 13772 log.cc:1196] T 13ed5317b3e54d89a8b3635ee75cfe81 P 64a56e7a8ca6492680dd530522f4a3f9: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-2/wal/wals/13ed5317b3e54d89a8b3635ee75cfe81
I20260806 02:45:57.303297 13774 ts_tablet_manager.cc:1939] T 13ed5317b3e54d89a8b3635ee75cfe81 P 0be05994eb164242b621cb037063e2d0: Deleting consensus metadata
I20260806 02:45:57.303453 13777 tablet_replica.cc:330] T 13ed5317b3e54d89a8b3635ee75cfe81 P 7d99f1b802814487b0455463222ed797: stopping tablet replica
I20260806 02:45:57.303797 13772 ts_tablet_manager.cc:1939] T 13ed5317b3e54d89a8b3635ee75cfe81 P 64a56e7a8ca6492680dd530522f4a3f9: Deleting consensus metadata
I20260806 02:45:57.303845 13777 raft_consensus.cc:2229] T 13ed5317b3e54d89a8b3635ee75cfe81 P 7d99f1b802814487b0455463222ed797 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:57.304232 13777 raft_consensus.cc:2258] T 13ed5317b3e54d89a8b3635ee75cfe81 P 7d99f1b802814487b0455463222ed797 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:57.304667 13599 tablet_service.cc:1467] Processing CreateTablet for tablet 5b7dc16c82624e5787949176a2c63825 (DEFAULT_TABLE table=test-table [id=4ff26683e3aa4029ab43c89c42d4bcc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:57.304787 13201 tablet_service.cc:1467] Processing CreateTablet for tablet 5b7dc16c82624e5787949176a2c63825 (DEFAULT_TABLE table=test-table [id=4ff26683e3aa4029ab43c89c42d4bcc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:57.304936 13599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b7dc16c82624e5787949176a2c63825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:57.305070 13201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b7dc16c82624e5787949176a2c63825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:57.305181 13404 tablet_service.cc:1467] Processing CreateTablet for tablet 5b7dc16c82624e5787949176a2c63825 (DEFAULT_TABLE table=test-table [id=4ff26683e3aa4029ab43c89c42d4bcc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:57.305421 13404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b7dc16c82624e5787949176a2c63825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:57.307312 13737 tablet_bootstrap.cc:492] T 5b7dc16c82624e5787949176a2c63825 P 64a56e7a8ca6492680dd530522f4a3f9: Bootstrap starting.
I20260806 02:45:57.307608 13738 tablet_bootstrap.cc:492] T 5b7dc16c82624e5787949176a2c63825 P 7d99f1b802814487b0455463222ed797: Bootstrap starting.
I20260806 02:45:57.307775 13735 tablet_bootstrap.cc:492] T 5b7dc16c82624e5787949176a2c63825 P 0be05994eb164242b621cb037063e2d0: Bootstrap starting.
I20260806 02:45:57.308400 13737 tablet_bootstrap.cc:654] T 5b7dc16c82624e5787949176a2c63825 P 64a56e7a8ca6492680dd530522f4a3f9: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:57.308696 13738 tablet_bootstrap.cc:654] T 5b7dc16c82624e5787949176a2c63825 P 7d99f1b802814487b0455463222ed797: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:57.308737 13735 tablet_bootstrap.cc:654] T 5b7dc16c82624e5787949176a2c63825 P 0be05994eb164242b621cb037063e2d0: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:57.309610 12979 catalog_manager.cc:4841] TS 0be05994eb164242b621cb037063e2d0 (127.1.183.130:44967): tablet 13ed5317b3e54d89a8b3635ee75cfe81 (table test-table [id=e622254399c540c1b6f0ee48dbb2e1b0]) successfully deleted
I20260806 02:45:57.310711 13737 tablet_bootstrap.cc:492] T 5b7dc16c82624e5787949176a2c63825 P 64a56e7a8ca6492680dd530522f4a3f9: No bootstrap required, opened a new log
I20260806 02:45:57.310808 13737 ts_tablet_manager.cc:1397] T 5b7dc16c82624e5787949176a2c63825 P 64a56e7a8ca6492680dd530522f4a3f9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 02:45:57.310909 12983 catalog_manager.cc:4841] TS 64a56e7a8ca6492680dd530522f4a3f9 (127.1.183.131:33073): tablet 13ed5317b3e54d89a8b3635ee75cfe81 (table test-table [id=e622254399c540c1b6f0ee48dbb2e1b0]) successfully deleted
I20260806 02:45:57.311295 13737 raft_consensus.cc:348] T 5b7dc16c82624e5787949176a2c63825 P 64a56e7a8ca6492680dd530522f4a3f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:57.311429 13737 raft_consensus.cc:374] T 5b7dc16c82624e5787949176a2c63825 P 64a56e7a8ca6492680dd530522f4a3f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:57.311462 13737 raft_consensus.cc:729] T 5b7dc16c82624e5787949176a2c63825 P 64a56e7a8ca6492680dd530522f4a3f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64a56e7a8ca6492680dd530522f4a3f9, State: Initialized, Role: FOLLOWER
I20260806 02:45:57.311590 13737 consensus_queue.cc:260] T 5b7dc16c82624e5787949176a2c63825 P 64a56e7a8ca6492680dd530522f4a3f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:57.311847 13737 ts_tablet_manager.cc:1428] T 5b7dc16c82624e5787949176a2c63825 P 64a56e7a8ca6492680dd530522f4a3f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:45:57.311962 13735 tablet_bootstrap.cc:492] T 5b7dc16c82624e5787949176a2c63825 P 0be05994eb164242b621cb037063e2d0: No bootstrap required, opened a new log
I20260806 02:45:57.312077 13735 ts_tablet_manager.cc:1397] T 5b7dc16c82624e5787949176a2c63825 P 0be05994eb164242b621cb037063e2d0: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 02:45:57.312650 13735 raft_consensus.cc:348] T 5b7dc16c82624e5787949176a2c63825 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:57.312772 13735 raft_consensus.cc:374] T 5b7dc16c82624e5787949176a2c63825 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:57.312817 13735 raft_consensus.cc:729] T 5b7dc16c82624e5787949176a2c63825 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0be05994eb164242b621cb037063e2d0, State: Initialized, Role: FOLLOWER
I20260806 02:45:57.312974 13735 consensus_queue.cc:260] T 5b7dc16c82624e5787949176a2c63825 P 0be05994eb164242b621cb037063e2d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:57.313270 13735 ts_tablet_manager.cc:1428] T 5b7dc16c82624e5787949176a2c63825 P 0be05994eb164242b621cb037063e2d0: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 02:45:57.313426 13738 tablet_bootstrap.cc:492] T 5b7dc16c82624e5787949176a2c63825 P 7d99f1b802814487b0455463222ed797: No bootstrap required, opened a new log
I20260806 02:45:57.313515 13738 ts_tablet_manager.cc:1397] T 5b7dc16c82624e5787949176a2c63825 P 7d99f1b802814487b0455463222ed797: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260806 02:45:57.313876 13777 ts_tablet_manager.cc:1905] T 13ed5317b3e54d89a8b3635ee75cfe81 P 7d99f1b802814487b0455463222ed797: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:57.314018 13738 raft_consensus.cc:348] T 5b7dc16c82624e5787949176a2c63825 P 7d99f1b802814487b0455463222ed797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:57.314141 13738 raft_consensus.cc:374] T 5b7dc16c82624e5787949176a2c63825 P 7d99f1b802814487b0455463222ed797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:57.314188 13738 raft_consensus.cc:729] T 5b7dc16c82624e5787949176a2c63825 P 7d99f1b802814487b0455463222ed797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d99f1b802814487b0455463222ed797, State: Initialized, Role: FOLLOWER
I20260806 02:45:57.314332 13738 consensus_queue.cc:260] T 5b7dc16c82624e5787949176a2c63825 P 7d99f1b802814487b0455463222ed797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:57.314599 13738 ts_tablet_manager.cc:1428] T 5b7dc16c82624e5787949176a2c63825 P 7d99f1b802814487b0455463222ed797: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:45:57.318238 13777 ts_tablet_manager.cc:1918] T 13ed5317b3e54d89a8b3635ee75cfe81 P 7d99f1b802814487b0455463222ed797: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:45:57.318333 13777 log.cc:1196] T 13ed5317b3e54d89a8b3635ee75cfe81 P 7d99f1b802814487b0455463222ed797: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-0/wal/wals/13ed5317b3e54d89a8b3635ee75cfe81
I20260806 02:45:57.318699 13777 ts_tablet_manager.cc:1939] T 13ed5317b3e54d89a8b3635ee75cfe81 P 7d99f1b802814487b0455463222ed797: Deleting consensus metadata
I20260806 02:45:57.319959 12983 catalog_manager.cc:4841] TS 7d99f1b802814487b0455463222ed797 (127.1.183.129:43601): tablet 13ed5317b3e54d89a8b3635ee75cfe81 (table test-table [id=e622254399c540c1b6f0ee48dbb2e1b0]) successfully deleted
W20260806 02:45:57.336974 13696 tablet.cc:2369] T 5b7dc16c82624e5787949176a2c63825 P 64a56e7a8ca6492680dd530522f4a3f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:45:57.341938 13496 tablet.cc:2369] T 5b7dc16c82624e5787949176a2c63825 P 0be05994eb164242b621cb037063e2d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:45:57.349812 13305 tablet.cc:2369] T 5b7dc16c82624e5787949176a2c63825 P 7d99f1b802814487b0455463222ed797: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:45:57.461638 13746 raft_consensus.cc:482] T 5b7dc16c82624e5787949176a2c63825 P 7d99f1b802814487b0455463222ed797 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:45:57.461831 13746 raft_consensus.cc:504] T 5b7dc16c82624e5787949176a2c63825 P 7d99f1b802814487b0455463222ed797 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:57.463164 13746 leader_election.cc:288] T 5b7dc16c82624e5787949176a2c63825 P 7d99f1b802814487b0455463222ed797 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0be05994eb164242b621cb037063e2d0 (127.1.183.130:44967), 64a56e7a8ca6492680dd530522f4a3f9 (127.1.183.131:33073)
I20260806 02:45:57.469174 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:57.463502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:44967 (local address 127.1.183.129:60069)
0806 02:45:57.464006 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:57.464021 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:57.464110 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:57.464423 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:57.464426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:57.464467 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:57.464799 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:57.464808 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.465893 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.465897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:57.467646 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:57.467658 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.468645 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.468649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:57.468769 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.468802 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 02:45:57.468893 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 02:45:57.468981 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":405,"spinlock_wait_cycles":11392,"thread_start_us":116,"threads_started":1}
I20260806 02:45:57.469184 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:57.463581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:60069 (local address 127.1.183.130:44967)
0806 02:45:57.463751 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:57.463757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:57.464113 (+ 356us) server_negotiation.cc:407] Connection header received
0806 02:45:57.464249 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:57.464252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:57.464303 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:57.464403 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:57.464943 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.465756 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.467815 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.468522 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.468657 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.468715 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:57.468931 (+ 216us) server_negotiation.cc:299] Negotiation successful
0806 02:45:57.469029 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:45:57.469363 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:57.463641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:45161 (local address 127.1.183.131:33073)
0806 02:45:57.463797 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:57.463802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:57.464088 (+ 286us) server_negotiation.cc:407] Connection header received
0806 02:45:57.464243 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:57.464246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:57.464303 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:57.464394 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:45:57.464961 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.465736 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.467843 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.468522 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.468657 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.468715 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:57.469136 (+ 421us) server_negotiation.cc:299] Negotiation successful
0806 02:45:57.469233 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:45:57.469774 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:57.463494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:33073 (local address 127.1.183.129:45161)
0806 02:45:57.463953 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:57.463969 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:57.464110 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:57.464423 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:57.464426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:57.464466 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:45:57.464799 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:57.464837 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.465893 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.465897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:57.467645 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:57.467657 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.468824 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.468827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:57.468979 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.469003 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 02:45:57.469569 (+ 566us) client_negotiation.cc:240] Negotiation successful
0806 02:45:57.469651 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":271,"spinlock_wait_cycles":21504,"thread_start_us":115,"threads_started":1}
I20260806 02:45:57.469972 13428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b7dc16c82624e5787949176a2c63825" candidate_uuid: "7d99f1b802814487b0455463222ed797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0be05994eb164242b621cb037063e2d0" is_pre_election: true
I20260806 02:45:57.470271 13428 raft_consensus.cc:2454] T 5b7dc16c82624e5787949176a2c63825 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d99f1b802814487b0455463222ed797 in term 0.
I20260806 02:45:57.470270 13627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b7dc16c82624e5787949176a2c63825" candidate_uuid: "7d99f1b802814487b0455463222ed797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64a56e7a8ca6492680dd530522f4a3f9" is_pre_election: true
I20260806 02:45:57.470420 13627 raft_consensus.cc:2454] T 5b7dc16c82624e5787949176a2c63825 P 64a56e7a8ca6492680dd530522f4a3f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d99f1b802814487b0455463222ed797 in term 0.
I20260806 02:45:57.470794 13138 leader_election.cc:302] T 5b7dc16c82624e5787949176a2c63825 P 7d99f1b802814487b0455463222ed797 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0be05994eb164242b621cb037063e2d0, 7d99f1b802814487b0455463222ed797; no voters:
I20260806 02:45:57.471078 13746 raft_consensus.cc:2781] T 5b7dc16c82624e5787949176a2c63825 P 7d99f1b802814487b0455463222ed797 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:45:57.471143 13746 raft_consensus.cc:482] T 5b7dc16c82624e5787949176a2c63825 P 7d99f1b802814487b0455463222ed797 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:45:57.471182 13746 raft_consensus.cc:3037] T 5b7dc16c82624e5787949176a2c63825 P 7d99f1b802814487b0455463222ed797 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:57.472296 13746 raft_consensus.cc:504] T 5b7dc16c82624e5787949176a2c63825 P 7d99f1b802814487b0455463222ed797 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:57.472681 13746 leader_election.cc:288] T 5b7dc16c82624e5787949176a2c63825 P 7d99f1b802814487b0455463222ed797 [CANDIDATE]: Term 1 election: Requested vote from peers 0be05994eb164242b621cb037063e2d0 (127.1.183.130:44967), 64a56e7a8ca6492680dd530522f4a3f9 (127.1.183.131:33073)
I20260806 02:45:57.473043 13428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b7dc16c82624e5787949176a2c63825" candidate_uuid: "7d99f1b802814487b0455463222ed797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0be05994eb164242b621cb037063e2d0"
I20260806 02:45:57.473117 13627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b7dc16c82624e5787949176a2c63825" candidate_uuid: "7d99f1b802814487b0455463222ed797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64a56e7a8ca6492680dd530522f4a3f9"
I20260806 02:45:57.473187 13428 raft_consensus.cc:3037] T 5b7dc16c82624e5787949176a2c63825 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:57.473232 13627 raft_consensus.cc:3037] T 5b7dc16c82624e5787949176a2c63825 P 64a56e7a8ca6492680dd530522f4a3f9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:57.474460 13627 raft_consensus.cc:2454] T 5b7dc16c82624e5787949176a2c63825 P 64a56e7a8ca6492680dd530522f4a3f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d99f1b802814487b0455463222ed797 in term 1.
I20260806 02:45:57.474471 13428 raft_consensus.cc:2454] T 5b7dc16c82624e5787949176a2c63825 P 0be05994eb164242b621cb037063e2d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d99f1b802814487b0455463222ed797 in term 1.
I20260806 02:45:57.474841 13140 leader_election.cc:302] T 5b7dc16c82624e5787949176a2c63825 P 7d99f1b802814487b0455463222ed797 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64a56e7a8ca6492680dd530522f4a3f9, 7d99f1b802814487b0455463222ed797; no voters:
I20260806 02:45:57.475044 13746 raft_consensus.cc:2781] T 5b7dc16c82624e5787949176a2c63825 P 7d99f1b802814487b0455463222ed797 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:57.475301 13746 raft_consensus.cc:686] T 5b7dc16c82624e5787949176a2c63825 P 7d99f1b802814487b0455463222ed797 [term 1 LEADER]: Becoming Leader. State: Replica: 7d99f1b802814487b0455463222ed797, State: Running, Role: LEADER
I20260806 02:45:57.475620 13746 consensus_queue.cc:237] T 5b7dc16c82624e5787949176a2c63825 P 7d99f1b802814487b0455463222ed797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:57.478193 13000 catalog_manager.cc:5495] T 5b7dc16c82624e5787949176a2c63825 P 7d99f1b802814487b0455463222ed797 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d99f1b802814487b0455463222ed797 (127.1.183.129). New cstate: current_term: 1 leader_uuid: "7d99f1b802814487b0455463222ed797" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } health_report { overall_health: UNKNOWN } } }
Aug 06 02:45:57 dist-test-slave-gmjp krb5kdc[12940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984356, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.190@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM
I20260806 02:45:57.543298 13069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:57.534170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48638 (local address 127.1.183.190:43847)
0806 02:45:57.534367 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:57.534373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:57.534557 (+ 184us) server_negotiation.cc:407] Connection header received
0806 02:45:57.534738 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:57.534741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:57.534796 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:57.534885 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:57.535611 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.536140 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.537074 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.537299 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.537411 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.539980 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:57.539984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:57.539986 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:57.540010 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:57.541522 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:57.542009 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:57.542011 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:57.542013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:57.542088 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:57.542389 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:57.542392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:57.542393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:57.542587 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:57.542667 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:57.543023 (+ 356us) server_negotiation.cc:299] Negotiation successful
0806 02:45:57.543167 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":65}
I20260806 02:45:57.543299 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:57.534055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:43847 (local address 127.0.0.1:48638)
0806 02:45:57.534446 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:57.534460 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:57.534624 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:57.534908 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:57.534910 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:57.535351 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:45:57.535448 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:57.535455 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.536313 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.536318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:57.536940 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:57.536949 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.537417 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.537421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:57.537497 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.538205 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:45:57.538219 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:45:57.539805 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:45:57.541637 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:57.541646 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:57.541653 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:57.541902 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:57.542197 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:45:57.542203 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:45:57.542207 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:45:57.542307 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:45:57.542683 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:45:57.542689 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:45:57.542855 (+ 166us) client_negotiation.cc:769] Sending connection context
0806 02:45:57.542969 (+ 114us) client_negotiation.cc:240] Negotiation successful
0806 02:45:57.543109 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":132,"threads_started":1}
I20260806 02:45:57.547724 13000 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:48638:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 02:45:57.548568 13000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:45:57.554025 13599 tablet_service.cc:1467] Processing CreateTablet for tablet 3f199ab1130a4e31b17b22a1862dc890 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2d9c55c6482c40c5a805bdd0f9960980]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:45:57.553988 13404 tablet_service.cc:1467] Processing CreateTablet for tablet 3f199ab1130a4e31b17b22a1862dc890 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2d9c55c6482c40c5a805bdd0f9960980]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:45:57.553988 13201 tablet_service.cc:1467] Processing CreateTablet for tablet 3f199ab1130a4e31b17b22a1862dc890 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2d9c55c6482c40c5a805bdd0f9960980]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:45:57.554456 13599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f199ab1130a4e31b17b22a1862dc890. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:57.554451 13404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f199ab1130a4e31b17b22a1862dc890. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:57.554451 13201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f199ab1130a4e31b17b22a1862dc890. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:57.557159 13737 tablet_bootstrap.cc:492] T 3f199ab1130a4e31b17b22a1862dc890 P 64a56e7a8ca6492680dd530522f4a3f9: Bootstrap starting.
I20260806 02:45:57.557449 13735 tablet_bootstrap.cc:492] T 3f199ab1130a4e31b17b22a1862dc890 P 0be05994eb164242b621cb037063e2d0: Bootstrap starting.
I20260806 02:45:57.557966 13738 tablet_bootstrap.cc:492] T 3f199ab1130a4e31b17b22a1862dc890 P 7d99f1b802814487b0455463222ed797: Bootstrap starting.
I20260806 02:45:57.558218 13737 tablet_bootstrap.cc:654] T 3f199ab1130a4e31b17b22a1862dc890 P 64a56e7a8ca6492680dd530522f4a3f9: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:57.558355 13735 tablet_bootstrap.cc:654] T 3f199ab1130a4e31b17b22a1862dc890 P 0be05994eb164242b621cb037063e2d0: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:57.559069 13738 tablet_bootstrap.cc:654] T 3f199ab1130a4e31b17b22a1862dc890 P 7d99f1b802814487b0455463222ed797: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:57.560501 13738 tablet_bootstrap.cc:492] T 3f199ab1130a4e31b17b22a1862dc890 P 7d99f1b802814487b0455463222ed797: No bootstrap required, opened a new log
I20260806 02:45:57.560604 13738 ts_tablet_manager.cc:1397] T 3f199ab1130a4e31b17b22a1862dc890 P 7d99f1b802814487b0455463222ed797: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 02:45:57.560828 13735 tablet_bootstrap.cc:492] T 3f199ab1130a4e31b17b22a1862dc890 P 0be05994eb164242b621cb037063e2d0: No bootstrap required, opened a new log
I20260806 02:45:57.560920 13735 ts_tablet_manager.cc:1397] T 3f199ab1130a4e31b17b22a1862dc890 P 0be05994eb164242b621cb037063e2d0: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 02:45:57.561134 13738 raft_consensus.cc:348] T 3f199ab1130a4e31b17b22a1862dc890 P 7d99f1b802814487b0455463222ed797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:57.561268 13738 raft_consensus.cc:374] T 3f199ab1130a4e31b17b22a1862dc890 P 7d99f1b802814487b0455463222ed797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:57.561317 13738 raft_consensus.cc:729] T 3f199ab1130a4e31b17b22a1862dc890 P 7d99f1b802814487b0455463222ed797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d99f1b802814487b0455463222ed797, State: Initialized, Role: FOLLOWER
I20260806 02:45:57.561319 13735 raft_consensus.cc:348] T 3f199ab1130a4e31b17b22a1862dc890 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:57.561440 13735 raft_consensus.cc:374] T 3f199ab1130a4e31b17b22a1862dc890 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:57.561480 13735 raft_consensus.cc:729] T 3f199ab1130a4e31b17b22a1862dc890 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0be05994eb164242b621cb037063e2d0, State: Initialized, Role: FOLLOWER
I20260806 02:45:57.561471 13738 consensus_queue.cc:260] T 3f199ab1130a4e31b17b22a1862dc890 P 7d99f1b802814487b0455463222ed797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:57.561609 13735 consensus_queue.cc:260] T 3f199ab1130a4e31b17b22a1862dc890 P 0be05994eb164242b621cb037063e2d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:57.561769 13738 ts_tablet_manager.cc:1428] T 3f199ab1130a4e31b17b22a1862dc890 P 7d99f1b802814487b0455463222ed797: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 02:45:57.561878 13735 ts_tablet_manager.cc:1428] T 3f199ab1130a4e31b17b22a1862dc890 P 0be05994eb164242b621cb037063e2d0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:45:57.562053 13742 tablet_replica.cc:439] T 3f199ab1130a4e31b17b22a1862dc890 P 0be05994eb164242b621cb037063e2d0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } } }
I20260806 02:45:57.562069 13746 tablet_replica.cc:439] T 3f199ab1130a4e31b17b22a1862dc890 P 7d99f1b802814487b0455463222ed797: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } } }
I20260806 02:45:57.562196 13742 tablet_replica.cc:442] T 3f199ab1130a4e31b17b22a1862dc890 P 0be05994eb164242b621cb037063e2d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:57.562196 13746 tablet_replica.cc:442] T 3f199ab1130a4e31b17b22a1862dc890 P 7d99f1b802814487b0455463222ed797: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:57.563146 13737 tablet_bootstrap.cc:492] T 3f199ab1130a4e31b17b22a1862dc890 P 64a56e7a8ca6492680dd530522f4a3f9: No bootstrap required, opened a new log
I20260806 02:45:57.563251 13737 ts_tablet_manager.cc:1397] T 3f199ab1130a4e31b17b22a1862dc890 P 64a56e7a8ca6492680dd530522f4a3f9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260806 02:45:57.563807 13737 raft_consensus.cc:348] T 3f199ab1130a4e31b17b22a1862dc890 P 64a56e7a8ca6492680dd530522f4a3f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:57.563973 13737 raft_consensus.cc:374] T 3f199ab1130a4e31b17b22a1862dc890 P 64a56e7a8ca6492680dd530522f4a3f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:57.564040 13737 raft_consensus.cc:729] T 3f199ab1130a4e31b17b22a1862dc890 P 64a56e7a8ca6492680dd530522f4a3f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64a56e7a8ca6492680dd530522f4a3f9, State: Initialized, Role: FOLLOWER
I20260806 02:45:57.564205 13737 consensus_queue.cc:260] T 3f199ab1130a4e31b17b22a1862dc890 P 64a56e7a8ca6492680dd530522f4a3f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:57.564505 13744 tablet_replica.cc:439] T 3f199ab1130a4e31b17b22a1862dc890 P 64a56e7a8ca6492680dd530522f4a3f9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } } }
I20260806 02:45:57.564656 13744 tablet_replica.cc:442] T 3f199ab1130a4e31b17b22a1862dc890 P 64a56e7a8ca6492680dd530522f4a3f9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:57.564893 13737 ts_tablet_manager.cc:1428] T 3f199ab1130a4e31b17b22a1862dc890 P 64a56e7a8ca6492680dd530522f4a3f9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:45:57.640820 13742 raft_consensus.cc:482] T 3f199ab1130a4e31b17b22a1862dc890 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:45:57.640977 13742 raft_consensus.cc:504] T 3f199ab1130a4e31b17b22a1862dc890 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:57.641443 13742 leader_election.cc:288] T 3f199ab1130a4e31b17b22a1862dc890 P 0be05994eb164242b621cb037063e2d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d99f1b802814487b0455463222ed797 (127.1.183.129:43601), 64a56e7a8ca6492680dd530522f4a3f9 (127.1.183.131:33073)
I20260806 02:45:57.641974 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f199ab1130a4e31b17b22a1862dc890" candidate_uuid: "0be05994eb164242b621cb037063e2d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d99f1b802814487b0455463222ed797" is_pre_election: true
I20260806 02:45:57.642040 13627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f199ab1130a4e31b17b22a1862dc890" candidate_uuid: "0be05994eb164242b621cb037063e2d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64a56e7a8ca6492680dd530522f4a3f9" is_pre_election: true
I20260806 02:45:57.642143 13234 raft_consensus.cc:2454] T 3f199ab1130a4e31b17b22a1862dc890 P 7d99f1b802814487b0455463222ed797 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0be05994eb164242b621cb037063e2d0 in term 0.
I20260806 02:45:57.642249 13627 raft_consensus.cc:2454] T 3f199ab1130a4e31b17b22a1862dc890 P 64a56e7a8ca6492680dd530522f4a3f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0be05994eb164242b621cb037063e2d0 in term 0.
I20260806 02:45:57.642567 13344 leader_election.cc:302] T 3f199ab1130a4e31b17b22a1862dc890 P 0be05994eb164242b621cb037063e2d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0be05994eb164242b621cb037063e2d0, 7d99f1b802814487b0455463222ed797; no voters:
I20260806 02:45:57.642805 13742 raft_consensus.cc:2781] T 3f199ab1130a4e31b17b22a1862dc890 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:45:57.642882 13742 raft_consensus.cc:482] T 3f199ab1130a4e31b17b22a1862dc890 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:45:57.642958 13742 raft_consensus.cc:3037] T 3f199ab1130a4e31b17b22a1862dc890 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:57.644014 13742 raft_consensus.cc:504] T 3f199ab1130a4e31b17b22a1862dc890 P 0be05994eb164242b621cb037063e2d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:57.644389 13742 leader_election.cc:288] T 3f199ab1130a4e31b17b22a1862dc890 P 0be05994eb164242b621cb037063e2d0 [CANDIDATE]: Term 1 election: Requested vote from peers 7d99f1b802814487b0455463222ed797 (127.1.183.129:43601), 64a56e7a8ca6492680dd530522f4a3f9 (127.1.183.131:33073)
I20260806 02:45:57.644805 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f199ab1130a4e31b17b22a1862dc890" candidate_uuid: "0be05994eb164242b621cb037063e2d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d99f1b802814487b0455463222ed797"
I20260806 02:45:57.644872 13627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f199ab1130a4e31b17b22a1862dc890" candidate_uuid: "0be05994eb164242b621cb037063e2d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64a56e7a8ca6492680dd530522f4a3f9"
I20260806 02:45:57.644966 13234 raft_consensus.cc:3037] T 3f199ab1130a4e31b17b22a1862dc890 P 7d99f1b802814487b0455463222ed797 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:57.645007 13627 raft_consensus.cc:3037] T 3f199ab1130a4e31b17b22a1862dc890 P 64a56e7a8ca6492680dd530522f4a3f9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:57.645987 13234 raft_consensus.cc:2454] T 3f199ab1130a4e31b17b22a1862dc890 P 7d99f1b802814487b0455463222ed797 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0be05994eb164242b621cb037063e2d0 in term 1.
I20260806 02:45:57.646015 13627 raft_consensus.cc:2454] T 3f199ab1130a4e31b17b22a1862dc890 P 64a56e7a8ca6492680dd530522f4a3f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0be05994eb164242b621cb037063e2d0 in term 1.
I20260806 02:45:57.646298 13344 leader_election.cc:302] T 3f199ab1130a4e31b17b22a1862dc890 P 0be05994eb164242b621cb037063e2d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0be05994eb164242b621cb037063e2d0, 7d99f1b802814487b0455463222ed797; no voters:
I20260806 02:45:57.646487 13742 raft_consensus.cc:2781] T 3f199ab1130a4e31b17b22a1862dc890 P 0be05994eb164242b621cb037063e2d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:57.646592 13742 raft_consensus.cc:686] T 3f199ab1130a4e31b17b22a1862dc890 P 0be05994eb164242b621cb037063e2d0 [term 1 LEADER]: Becoming Leader. State: Replica: 0be05994eb164242b621cb037063e2d0, State: Running, Role: LEADER
I20260806 02:45:57.646732 13742 consensus_queue.cc:237] T 3f199ab1130a4e31b17b22a1862dc890 P 0be05994eb164242b621cb037063e2d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:57.647436 13749 tablet_replica.cc:439] T 3f199ab1130a4e31b17b22a1862dc890 P 0be05994eb164242b621cb037063e2d0: TxnStatusTablet state changed. Reason: New leader 0be05994eb164242b621cb037063e2d0. Latest consensus state: current_term: 1 leader_uuid: "0be05994eb164242b621cb037063e2d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } } }
I20260806 02:45:57.647555 13749 tablet_replica.cc:442] T 3f199ab1130a4e31b17b22a1862dc890 P 0be05994eb164242b621cb037063e2d0: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:45:57.647794 13802 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 02:45:57.647969 13802 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 02:45:57.648675 13000 catalog_manager.cc:5495] T 3f199ab1130a4e31b17b22a1862dc890 P 0be05994eb164242b621cb037063e2d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0be05994eb164242b621cb037063e2d0 (127.1.183.130). New cstate: current_term: 1 leader_uuid: "0be05994eb164242b621cb037063e2d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } health_report { overall_health: UNKNOWN } } }
I20260806 02:45:57.670056 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:57.666293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43528 (local address 127.1.183.130:44967)
0806 02:45:57.666445 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:57.666450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:57.666464 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:45:57.666635 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:57.666638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:57.666714 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:57.666812 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:57.667122 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.667629 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.668954 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.669168 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.669244 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.669595 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:57.669669 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:57.669718 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:57.669866 (+ 148us) server_negotiation.cc:299] Negotiation successful
0806 02:45:57.669935 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 02:45:57.670094 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:57.666157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:44967 (local address 127.0.0.1:43528)
0806 02:45:57.666330 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:57.666350 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:57.666480 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:57.666850 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:57.666854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:57.666864 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:45:57.667004 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:57.667010 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.667727 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.667730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:57.668752 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:57.668763 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.669285 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.669289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:57.669386 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.669448 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:45:57.669753 (+ 305us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:45:57.669769 (+ 16us) client_negotiation.cc:769] Sending connection context
0806 02:45:57.669848 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 02:45:57.669943 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260806 02:45:57.672791 13234 raft_consensus.cc:1261] T 3f199ab1130a4e31b17b22a1862dc890 P 7d99f1b802814487b0455463222ed797 [term 1 FOLLOWER]: Refusing update from remote peer 0be05994eb164242b621cb037063e2d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:57.672844 13627 raft_consensus.cc:1261] T 3f199ab1130a4e31b17b22a1862dc890 P 64a56e7a8ca6492680dd530522f4a3f9 [term 1 FOLLOWER]: Refusing update from remote peer 0be05994eb164242b621cb037063e2d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:57.673311 13749 consensus_queue.cc:1035] T 3f199ab1130a4e31b17b22a1862dc890 P 0be05994eb164242b621cb037063e2d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:57.673492 13742 consensus_queue.cc:1035] T 3f199ab1130a4e31b17b22a1862dc890 P 0be05994eb164242b621cb037063e2d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:57.674959 13744 tablet_replica.cc:439] T 3f199ab1130a4e31b17b22a1862dc890 P 64a56e7a8ca6492680dd530522f4a3f9: TxnStatusTablet state changed. Reason: New leader 0be05994eb164242b621cb037063e2d0. Latest consensus state: current_term: 1 leader_uuid: "0be05994eb164242b621cb037063e2d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } } }
I20260806 02:45:57.675078 13744 tablet_replica.cc:442] T 3f199ab1130a4e31b17b22a1862dc890 P 64a56e7a8ca6492680dd530522f4a3f9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:57.675145 13746 tablet_replica.cc:439] T 3f199ab1130a4e31b17b22a1862dc890 P 7d99f1b802814487b0455463222ed797: TxnStatusTablet state changed. Reason: New leader 0be05994eb164242b621cb037063e2d0. Latest consensus state: current_term: 1 leader_uuid: "0be05994eb164242b621cb037063e2d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } } }
I20260806 02:45:57.675263 13746 tablet_replica.cc:442] T 3f199ab1130a4e31b17b22a1862dc890 P 7d99f1b802814487b0455463222ed797: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:57.676462 13749 tablet_replica.cc:439] T 3f199ab1130a4e31b17b22a1862dc890 P 0be05994eb164242b621cb037063e2d0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0be05994eb164242b621cb037063e2d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } } }
I20260806 02:45:57.676574 13749 tablet_replica.cc:442] T 3f199ab1130a4e31b17b22a1862dc890 P 0be05994eb164242b621cb037063e2d0: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:45:57.676680 13756 mvcc.cc:205] Tried to move back new op lower bound from 7315391929023041536 to 7315391928922472448. Current Snapshot: MvccSnapshot[applied={T|T < 7315391929023041536}]
I20260806 02:45:57.678192 13744 tablet_replica.cc:439] T 3f199ab1130a4e31b17b22a1862dc890 P 64a56e7a8ca6492680dd530522f4a3f9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0be05994eb164242b621cb037063e2d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } } }
I20260806 02:45:57.678325 13744 tablet_replica.cc:442] T 3f199ab1130a4e31b17b22a1862dc890 P 64a56e7a8ca6492680dd530522f4a3f9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:57.679296 13742 tablet_replica.cc:439] T 3f199ab1130a4e31b17b22a1862dc890 P 0be05994eb164242b621cb037063e2d0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0be05994eb164242b621cb037063e2d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } } }
I20260806 02:45:57.679412 13742 tablet_replica.cc:442] T 3f199ab1130a4e31b17b22a1862dc890 P 0be05994eb164242b621cb037063e2d0: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:45:57.681586 13746 tablet_replica.cc:439] T 3f199ab1130a4e31b17b22a1862dc890 P 7d99f1b802814487b0455463222ed797: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0be05994eb164242b621cb037063e2d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } } }
I20260806 02:45:57.681743 13746 tablet_replica.cc:442] T 3f199ab1130a4e31b17b22a1862dc890 P 7d99f1b802814487b0455463222ed797: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:45:57.688827 13734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:57.685136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35042 (local address 127.1.183.129:43601)
0806 02:45:57.685282 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:57.685287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:57.685300 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:45:57.685465 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:57.685468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:57.685524 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:57.685638 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:57.685884 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.686400 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.687237 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.687471 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.687579 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.687879 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:57.688429 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:57.688558 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:57.688619 (+ 61us) server_negotiation.cc:299] Negotiation successful
0806 02:45:57.688678 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:45:57.700291 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:57.696525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:35359 (local address 127.1.183.130:44967)
0806 02:45:57.696667 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:57.696671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:57.696826 (+ 155us) server_negotiation.cc:407] Connection header received
0806 02:45:57.696975 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:57.696977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:57.697024 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:57.697088 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:57.697362 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.697860 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.699162 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.699371 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.699454 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.699786 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:57.699859 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:57.699920 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:57.700044 (+ 124us) server_negotiation.cc:299] Negotiation successful
0806 02:45:57.700098 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:45:57.700335 13809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:57.696367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:44967 (local address 127.1.183.129:35359)
0806 02:45:57.696722 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:57.696738 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:57.696860 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:57.697118 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:57.697121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:57.697129 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:45:57.697259 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:57.697266 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.697960 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.697963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:57.699037 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:57.699049 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.699480 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.699484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:57.699586 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.699648 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:45:57.699946 (+ 298us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:45:57.699953 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 02:45:57.700022 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 02:45:57.700108 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":145,"threads_started":1}
I20260806 02:45:57.708623 13627 raft_consensus.cc:1261] T 5b7dc16c82624e5787949176a2c63825 P 64a56e7a8ca6492680dd530522f4a3f9 [term 1 FOLLOWER]: Refusing update from remote peer 7d99f1b802814487b0455463222ed797: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:57.708827 13428 raft_consensus.cc:1261] T 5b7dc16c82624e5787949176a2c63825 P 0be05994eb164242b621cb037063e2d0 [term 1 FOLLOWER]: Refusing update from remote peer 7d99f1b802814487b0455463222ed797: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:57.709262 13746 consensus_queue.cc:1035] T 5b7dc16c82624e5787949176a2c63825 P 7d99f1b802814487b0455463222ed797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:57.709462 13788 consensus_queue.cc:1035] T 5b7dc16c82624e5787949176a2c63825 P 7d99f1b802814487b0455463222ed797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:57.734052 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:57.729903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:43601 (local address 127.1.183.130:49767)
0806 02:45:57.730316 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:57.730331 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:57.730444 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:57.730684 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:57.730687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:57.730695 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:45:57.730863 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:57.730869 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.731804 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.731810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:57.732768 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:57.732782 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.733365 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.733368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:57.733445 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.733497 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:45:57.733806 (+ 309us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:45:57.733810 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 02:45:57.733852 (+ 42us) client_negotiation.cc:240] Negotiation successful
0806 02:45:57.733921 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":119,"threads_started":1}
I20260806 02:45:57.734061 13734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:57.729998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:49767 (local address 127.1.183.129:43601)
0806 02:45:57.730113 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:57.730118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:57.730396 (+ 278us) server_negotiation.cc:407] Connection header received
0806 02:45:57.730534 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:57.730537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:57.730583 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:57.730645 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:57.730972 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.731685 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.732903 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.733250 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.733355 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.733635 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:57.733722 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:57.733809 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:57.733864 (+ 55us) server_negotiation.cc:299] Negotiation successful
0806 02:45:57.733924 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 02:45:57.761742 12997 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:57.761930 12997 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:57.764108 12997 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1557f14f7b246a3aee81db7b9d5a7eb: Sending DeleteTablet for 3 replicas of tablet 5b7dc16c82624e5787949176a2c63825
I20260806 02:45:57.764780 13404 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b7dc16c82624e5787949176a2c63825 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:57 UTC) from {username='oryx', principal='oryx/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:43500
I20260806 02:45:57.764782 13201 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b7dc16c82624e5787949176a2c63825 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:57 UTC) from {username='oryx', principal='oryx/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:35030
I20260806 02:45:57.764997 13774 tablet_replica.cc:330] T 5b7dc16c82624e5787949176a2c63825 P 0be05994eb164242b621cb037063e2d0: stopping tablet replica
I20260806 02:45:57.765125 13774 raft_consensus.cc:2229] T 5b7dc16c82624e5787949176a2c63825 P 0be05994eb164242b621cb037063e2d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:57.765256 13774 raft_consensus.cc:2258] T 5b7dc16c82624e5787949176a2c63825 P 0be05994eb164242b621cb037063e2d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:57.765434 13777 tablet_replica.cc:330] T 5b7dc16c82624e5787949176a2c63825 P 7d99f1b802814487b0455463222ed797: stopping tablet replica
I20260806 02:45:57.765547 13777 raft_consensus.cc:2229] T 5b7dc16c82624e5787949176a2c63825 P 7d99f1b802814487b0455463222ed797 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:45:57.765885 13777 raft_consensus.cc:2258] T 5b7dc16c82624e5787949176a2c63825 P 7d99f1b802814487b0455463222ed797 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:57.766574 13599 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b7dc16c82624e5787949176a2c63825 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:57 UTC) from {username='oryx', principal='oryx/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:37104
I20260806 02:45:57.766688 13777 ts_tablet_manager.cc:1905] T 5b7dc16c82624e5787949176a2c63825 P 7d99f1b802814487b0455463222ed797: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:57.766772 13772 tablet_replica.cc:330] T 5b7dc16c82624e5787949176a2c63825 P 64a56e7a8ca6492680dd530522f4a3f9: stopping tablet replica
I20260806 02:45:57.766873 13772 raft_consensus.cc:2229] T 5b7dc16c82624e5787949176a2c63825 P 64a56e7a8ca6492680dd530522f4a3f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:57.766999 13772 raft_consensus.cc:2258] T 5b7dc16c82624e5787949176a2c63825 P 64a56e7a8ca6492680dd530522f4a3f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:57.768137 13774 ts_tablet_manager.cc:1905] T 5b7dc16c82624e5787949176a2c63825 P 0be05994eb164242b621cb037063e2d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:57.769871 13777 ts_tablet_manager.cc:1918] T 5b7dc16c82624e5787949176a2c63825 P 7d99f1b802814487b0455463222ed797: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:45:57.769944 13777 log.cc:1196] T 5b7dc16c82624e5787949176a2c63825 P 7d99f1b802814487b0455463222ed797: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-0/wal/wals/5b7dc16c82624e5787949176a2c63825
I20260806 02:45:57.770236 13777 ts_tablet_manager.cc:1939] T 5b7dc16c82624e5787949176a2c63825 P 7d99f1b802814487b0455463222ed797: Deleting consensus metadata
I20260806 02:45:57.771061 12983 catalog_manager.cc:4841] TS 7d99f1b802814487b0455463222ed797 (127.1.183.129:43601): tablet 5b7dc16c82624e5787949176a2c63825 (table test-table [id=4ff26683e3aa4029ab43c89c42d4bcc1]) successfully deleted
I20260806 02:45:57.772214 13774 ts_tablet_manager.cc:1918] T 5b7dc16c82624e5787949176a2c63825 P 0be05994eb164242b621cb037063e2d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:45:57.772379 13774 log.cc:1196] T 5b7dc16c82624e5787949176a2c63825 P 0be05994eb164242b621cb037063e2d0: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-1/wal/wals/5b7dc16c82624e5787949176a2c63825
I20260806 02:45:57.772795 13774 ts_tablet_manager.cc:1939] T 5b7dc16c82624e5787949176a2c63825 P 0be05994eb164242b621cb037063e2d0: Deleting consensus metadata
I20260806 02:45:57.773679 13772 ts_tablet_manager.cc:1905] T 5b7dc16c82624e5787949176a2c63825 P 64a56e7a8ca6492680dd530522f4a3f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:57.776765 13772 ts_tablet_manager.cc:1918] T 5b7dc16c82624e5787949176a2c63825 P 64a56e7a8ca6492680dd530522f4a3f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:45:57.776829 13772 log.cc:1196] T 5b7dc16c82624e5787949176a2c63825 P 64a56e7a8ca6492680dd530522f4a3f9: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-2/wal/wals/5b7dc16c82624e5787949176a2c63825
I20260806 02:45:57.777024 13772 ts_tablet_manager.cc:1939] T 5b7dc16c82624e5787949176a2c63825 P 64a56e7a8ca6492680dd530522f4a3f9: Deleting consensus metadata
I20260806 02:45:57.777897 12983 catalog_manager.cc:4841] TS 64a56e7a8ca6492680dd530522f4a3f9 (127.1.183.131:33073): tablet 5b7dc16c82624e5787949176a2c63825 (table test-table [id=4ff26683e3aa4029ab43c89c42d4bcc1]) successfully deleted
I20260806 02:45:57.777897 12979 catalog_manager.cc:4841] TS 0be05994eb164242b621cb037063e2d0 (127.1.183.130:44967): tablet 5b7dc16c82624e5787949176a2c63825 (table test-table [id=4ff26683e3aa4029ab43c89c42d4bcc1]) successfully deleted
Aug 06 02:45:57 dist-test-slave-gmjp krb5kdc[12940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:45:57 dist-test-slave-gmjp krb5kdc[12940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:45:57 dist-test-slave-gmjp krb5kdc[12940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 02:45:57.805970 13069 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:45:57.798705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48644 (local address 127.1.183.190:43847)
0806 02:45:57.798851 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:57.798856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:57.799150 (+ 294us) server_negotiation.cc:407] Connection header received
0806 02:45:57.799320 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:57.799328 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:57.799375 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:57.799467 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:57.800316 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.800853 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.801804 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.802145 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.802259 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.805849 (+ 3590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48644: BlockingRecv error: recv got EOF from 127.0.0.1:48644 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 06 02:45:57 dist-test-slave-gmjp krb5kdc[12940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM
I20260806 02:45:57.817241 13069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:57.808331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48652 (local address 127.1.183.190:43847)
0806 02:45:57.808606 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:57.808610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:57.808625 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:45:57.808730 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:57.808733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:57.808785 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:57.808870 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:45:57.809569 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.810093 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.810961 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:57.811184 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:57.811296 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:57.813875 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:45:57.813881 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:45:57.813889 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:45:57.813922 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:45:57.815452 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:57.815884 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:57.815887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:57.815889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:57.815980 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:45:57.816315 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:45:57.816319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:45:57.816321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:45:57.816519 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:45:57.816665 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:57.816920 (+ 255us) server_negotiation.cc:299] Negotiation successful
0806 02:45:57.817047 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260806 02:45:57.820086 12997 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:45:57.820655 12997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:45:57.825878 13201 tablet_service.cc:1467] Processing CreateTablet for tablet c73a9404e32243139a524b6171daebd7 (DEFAULT_TABLE table=test-table [id=bd6fa72f58d24b108e1a83f3648061ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:57.826248 13201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c73a9404e32243139a524b6171daebd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:57.826344 13404 tablet_service.cc:1467] Processing CreateTablet for tablet c73a9404e32243139a524b6171daebd7 (DEFAULT_TABLE table=test-table [id=bd6fa72f58d24b108e1a83f3648061ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:57.826643 13404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c73a9404e32243139a524b6171daebd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:57.829085 13599 tablet_service.cc:1467] Processing CreateTablet for tablet c73a9404e32243139a524b6171daebd7 (DEFAULT_TABLE table=test-table [id=bd6fa72f58d24b108e1a83f3648061ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:57.829384 13599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c73a9404e32243139a524b6171daebd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:57.829644 13735 tablet_bootstrap.cc:492] T c73a9404e32243139a524b6171daebd7 P 0be05994eb164242b621cb037063e2d0: Bootstrap starting.
I20260806 02:45:57.830502 13735 tablet_bootstrap.cc:654] T c73a9404e32243139a524b6171daebd7 P 0be05994eb164242b621cb037063e2d0: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:57.831586 13738 tablet_bootstrap.cc:492] T c73a9404e32243139a524b6171daebd7 P 7d99f1b802814487b0455463222ed797: Bootstrap starting.
I20260806 02:45:57.831893 13737 tablet_bootstrap.cc:492] T c73a9404e32243139a524b6171daebd7 P 64a56e7a8ca6492680dd530522f4a3f9: Bootstrap starting.
I20260806 02:45:57.832618 13738 tablet_bootstrap.cc:654] T c73a9404e32243139a524b6171daebd7 P 7d99f1b802814487b0455463222ed797: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:57.832892 13737 tablet_bootstrap.cc:654] T c73a9404e32243139a524b6171daebd7 P 64a56e7a8ca6492680dd530522f4a3f9: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:57.833153 13735 tablet_bootstrap.cc:492] T c73a9404e32243139a524b6171daebd7 P 0be05994eb164242b621cb037063e2d0: No bootstrap required, opened a new log
I20260806 02:45:57.833240 13735 ts_tablet_manager.cc:1397] T c73a9404e32243139a524b6171daebd7 P 0be05994eb164242b621cb037063e2d0: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260806 02:45:57.833667 13735 raft_consensus.cc:348] T c73a9404e32243139a524b6171daebd7 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:57.833791 13735 raft_consensus.cc:374] T c73a9404e32243139a524b6171daebd7 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:57.833835 13735 raft_consensus.cc:729] T c73a9404e32243139a524b6171daebd7 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0be05994eb164242b621cb037063e2d0, State: Initialized, Role: FOLLOWER
I20260806 02:45:57.833877 13738 tablet_bootstrap.cc:492] T c73a9404e32243139a524b6171daebd7 P 7d99f1b802814487b0455463222ed797: No bootstrap required, opened a new log
I20260806 02:45:57.833963 13738 ts_tablet_manager.cc:1397] T c73a9404e32243139a524b6171daebd7 P 7d99f1b802814487b0455463222ed797: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260806 02:45:57.833982 13735 consensus_queue.cc:260] T c73a9404e32243139a524b6171daebd7 P 0be05994eb164242b621cb037063e2d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:57.834266 13735 ts_tablet_manager.cc:1428] T c73a9404e32243139a524b6171daebd7 P 0be05994eb164242b621cb037063e2d0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:45:57.834400 13738 raft_consensus.cc:348] T c73a9404e32243139a524b6171daebd7 P 7d99f1b802814487b0455463222ed797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:57.834512 13738 raft_consensus.cc:374] T c73a9404e32243139a524b6171daebd7 P 7d99f1b802814487b0455463222ed797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:57.834543 13738 raft_consensus.cc:729] T c73a9404e32243139a524b6171daebd7 P 7d99f1b802814487b0455463222ed797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d99f1b802814487b0455463222ed797, State: Initialized, Role: FOLLOWER
I20260806 02:45:57.834661 13738 consensus_queue.cc:260] T c73a9404e32243139a524b6171daebd7 P 7d99f1b802814487b0455463222ed797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:57.834932 13738 ts_tablet_manager.cc:1428] T c73a9404e32243139a524b6171daebd7 P 7d99f1b802814487b0455463222ed797: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:45:57.836069 13737 tablet_bootstrap.cc:492] T c73a9404e32243139a524b6171daebd7 P 64a56e7a8ca6492680dd530522f4a3f9: No bootstrap required, opened a new log
I20260806 02:45:57.836190 13737 ts_tablet_manager.cc:1397] T c73a9404e32243139a524b6171daebd7 P 64a56e7a8ca6492680dd530522f4a3f9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 02:45:57.836669 13737 raft_consensus.cc:348] T c73a9404e32243139a524b6171daebd7 P 64a56e7a8ca6492680dd530522f4a3f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:57.836779 13737 raft_consensus.cc:374] T c73a9404e32243139a524b6171daebd7 P 64a56e7a8ca6492680dd530522f4a3f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:57.836812 13737 raft_consensus.cc:729] T c73a9404e32243139a524b6171daebd7 P 64a56e7a8ca6492680dd530522f4a3f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64a56e7a8ca6492680dd530522f4a3f9, State: Initialized, Role: FOLLOWER
I20260806 02:45:57.836930 13737 consensus_queue.cc:260] T c73a9404e32243139a524b6171daebd7 P 64a56e7a8ca6492680dd530522f4a3f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:57.837198 13737 ts_tablet_manager.cc:1428] T c73a9404e32243139a524b6171daebd7 P 64a56e7a8ca6492680dd530522f4a3f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 02:45:58.187201 13814 raft_consensus.cc:482] T c73a9404e32243139a524b6171daebd7 P 7d99f1b802814487b0455463222ed797 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:45:58.187390 13814 raft_consensus.cc:504] T c73a9404e32243139a524b6171daebd7 P 7d99f1b802814487b0455463222ed797 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:58.187947 13814 leader_election.cc:288] T c73a9404e32243139a524b6171daebd7 P 7d99f1b802814487b0455463222ed797 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0be05994eb164242b621cb037063e2d0 (127.1.183.130:44967), 64a56e7a8ca6492680dd530522f4a3f9 (127.1.183.131:33073)
I20260806 02:45:58.188577 13627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73a9404e32243139a524b6171daebd7" candidate_uuid: "7d99f1b802814487b0455463222ed797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64a56e7a8ca6492680dd530522f4a3f9" is_pre_election: true
I20260806 02:45:58.188786 13627 raft_consensus.cc:2454] T c73a9404e32243139a524b6171daebd7 P 64a56e7a8ca6492680dd530522f4a3f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d99f1b802814487b0455463222ed797 in term 0.
I20260806 02:45:58.188769 13428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73a9404e32243139a524b6171daebd7" candidate_uuid: "7d99f1b802814487b0455463222ed797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0be05994eb164242b621cb037063e2d0" is_pre_election: true
I20260806 02:45:58.188917 13428 raft_consensus.cc:2454] T c73a9404e32243139a524b6171daebd7 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d99f1b802814487b0455463222ed797 in term 0.
I20260806 02:45:58.189219 13140 leader_election.cc:302] T c73a9404e32243139a524b6171daebd7 P 7d99f1b802814487b0455463222ed797 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0be05994eb164242b621cb037063e2d0, 64a56e7a8ca6492680dd530522f4a3f9, 7d99f1b802814487b0455463222ed797; no voters:
I20260806 02:45:58.189396 13814 raft_consensus.cc:2781] T c73a9404e32243139a524b6171daebd7 P 7d99f1b802814487b0455463222ed797 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:45:58.189479 13814 raft_consensus.cc:482] T c73a9404e32243139a524b6171daebd7 P 7d99f1b802814487b0455463222ed797 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:45:58.189512 13814 raft_consensus.cc:3037] T c73a9404e32243139a524b6171daebd7 P 7d99f1b802814487b0455463222ed797 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:58.190691 13814 raft_consensus.cc:504] T c73a9404e32243139a524b6171daebd7 P 7d99f1b802814487b0455463222ed797 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:58.191056 13814 leader_election.cc:288] T c73a9404e32243139a524b6171daebd7 P 7d99f1b802814487b0455463222ed797 [CANDIDATE]: Term 1 election: Requested vote from peers 0be05994eb164242b621cb037063e2d0 (127.1.183.130:44967), 64a56e7a8ca6492680dd530522f4a3f9 (127.1.183.131:33073)
I20260806 02:45:58.191468 13428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73a9404e32243139a524b6171daebd7" candidate_uuid: "7d99f1b802814487b0455463222ed797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0be05994eb164242b621cb037063e2d0"
I20260806 02:45:58.191531 13627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c73a9404e32243139a524b6171daebd7" candidate_uuid: "7d99f1b802814487b0455463222ed797" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64a56e7a8ca6492680dd530522f4a3f9"
I20260806 02:45:58.191627 13428 raft_consensus.cc:3037] T c73a9404e32243139a524b6171daebd7 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:58.191753 13627 raft_consensus.cc:3037] T c73a9404e32243139a524b6171daebd7 P 64a56e7a8ca6492680dd530522f4a3f9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:58.192782 13428 raft_consensus.cc:2454] T c73a9404e32243139a524b6171daebd7 P 0be05994eb164242b621cb037063e2d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d99f1b802814487b0455463222ed797 in term 1.
I20260806 02:45:58.193003 13627 raft_consensus.cc:2454] T c73a9404e32243139a524b6171daebd7 P 64a56e7a8ca6492680dd530522f4a3f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d99f1b802814487b0455463222ed797 in term 1.
I20260806 02:45:58.193161 13138 leader_election.cc:302] T c73a9404e32243139a524b6171daebd7 P 7d99f1b802814487b0455463222ed797 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0be05994eb164242b621cb037063e2d0, 7d99f1b802814487b0455463222ed797; no voters:
I20260806 02:45:58.193377 13814 raft_consensus.cc:2781] T c73a9404e32243139a524b6171daebd7 P 7d99f1b802814487b0455463222ed797 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:58.193491 13814 raft_consensus.cc:686] T c73a9404e32243139a524b6171daebd7 P 7d99f1b802814487b0455463222ed797 [term 1 LEADER]: Becoming Leader. State: Replica: 7d99f1b802814487b0455463222ed797, State: Running, Role: LEADER
I20260806 02:45:58.193646 13814 consensus_queue.cc:237] T c73a9404e32243139a524b6171daebd7 P 7d99f1b802814487b0455463222ed797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:58.195411 12997 catalog_manager.cc:5495] T c73a9404e32243139a524b6171daebd7 P 7d99f1b802814487b0455463222ed797 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d99f1b802814487b0455463222ed797 (127.1.183.129). New cstate: current_term: 1 leader_uuid: "7d99f1b802814487b0455463222ed797" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } health_report { overall_health: UNKNOWN } } }
I20260806 02:45:58.205278 13734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:58.201798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35052 (local address 127.1.183.129:43601)
0806 02:45:58.201940 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:58.201945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:58.201981 (+ 36us) server_negotiation.cc:407] Connection header received
0806 02:45:58.202072 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:58.202075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:58.202129 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:58.202213 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:58.202480 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:58.203139 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:58.204099 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:58.204425 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:58.204538 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:58.204808 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:58.204895 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:58.204976 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:58.205073 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 02:45:58.205131 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 02:45:58.207545 13428 raft_consensus.cc:1261] T c73a9404e32243139a524b6171daebd7 P 0be05994eb164242b621cb037063e2d0 [term 1 FOLLOWER]: Refusing update from remote peer 7d99f1b802814487b0455463222ed797: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:58.207589 13627 raft_consensus.cc:1261] T c73a9404e32243139a524b6171daebd7 P 64a56e7a8ca6492680dd530522f4a3f9 [term 1 FOLLOWER]: Refusing update from remote peer 7d99f1b802814487b0455463222ed797: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:58.207999 13814 consensus_queue.cc:1035] T c73a9404e32243139a524b6171daebd7 P 7d99f1b802814487b0455463222ed797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:58.208220 13788 consensus_queue.cc:1035] T c73a9404e32243139a524b6171daebd7 P 7d99f1b802814487b0455463222ed797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:58.217386 12994 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:58.217545 12994 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:58.219702 12994 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1557f14f7b246a3aee81db7b9d5a7eb: Sending DeleteTablet for 3 replicas of tablet c73a9404e32243139a524b6171daebd7
I20260806 02:45:58.220378 13201 tablet_service.cc:1514] Processing DeleteTablet for tablet c73a9404e32243139a524b6171daebd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:58 UTC) from {username='oryx', principal='oryx/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:35030
I20260806 02:45:58.220458 13599 tablet_service.cc:1514] Processing DeleteTablet for tablet c73a9404e32243139a524b6171daebd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:58 UTC) from {username='oryx', principal='oryx/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:37104
I20260806 02:45:58.220458 13404 tablet_service.cc:1514] Processing DeleteTablet for tablet c73a9404e32243139a524b6171daebd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:58 UTC) from {username='oryx', principal='oryx/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:43500
I20260806 02:45:58.220608 13777 tablet_replica.cc:330] T c73a9404e32243139a524b6171daebd7 P 7d99f1b802814487b0455463222ed797: stopping tablet replica
I20260806 02:45:58.220643 13772 tablet_replica.cc:330] T c73a9404e32243139a524b6171daebd7 P 64a56e7a8ca6492680dd530522f4a3f9: stopping tablet replica
I20260806 02:45:58.220660 13774 tablet_replica.cc:330] T c73a9404e32243139a524b6171daebd7 P 0be05994eb164242b621cb037063e2d0: stopping tablet replica
I20260806 02:45:58.220738 13772 raft_consensus.cc:2229] T c73a9404e32243139a524b6171daebd7 P 64a56e7a8ca6492680dd530522f4a3f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:58.220739 13777 raft_consensus.cc:2229] T c73a9404e32243139a524b6171daebd7 P 7d99f1b802814487b0455463222ed797 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:45:58.220778 13774 raft_consensus.cc:2229] T c73a9404e32243139a524b6171daebd7 P 0be05994eb164242b621cb037063e2d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:58.220907 13772 raft_consensus.cc:2258] T c73a9404e32243139a524b6171daebd7 P 64a56e7a8ca6492680dd530522f4a3f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:58.220923 13774 raft_consensus.cc:2258] T c73a9404e32243139a524b6171daebd7 P 0be05994eb164242b621cb037063e2d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:58.220986 13777 raft_consensus.cc:2258] T c73a9404e32243139a524b6171daebd7 P 7d99f1b802814487b0455463222ed797 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:58.221697 13777 ts_tablet_manager.cc:1905] T c73a9404e32243139a524b6171daebd7 P 7d99f1b802814487b0455463222ed797: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:58.221697 13772 ts_tablet_manager.cc:1905] T c73a9404e32243139a524b6171daebd7 P 64a56e7a8ca6492680dd530522f4a3f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:58.221725 13774 ts_tablet_manager.cc:1905] T c73a9404e32243139a524b6171daebd7 P 0be05994eb164242b621cb037063e2d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:58.222007 12994 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:45:58.222630 12994 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:45:58.224349 13772 ts_tablet_manager.cc:1918] T c73a9404e32243139a524b6171daebd7 P 64a56e7a8ca6492680dd530522f4a3f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:45:58.224342 13777 ts_tablet_manager.cc:1918] T c73a9404e32243139a524b6171daebd7 P 7d99f1b802814487b0455463222ed797: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:45:58.224431 13777 log.cc:1196] T c73a9404e32243139a524b6171daebd7 P 7d99f1b802814487b0455463222ed797: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-0/wal/wals/c73a9404e32243139a524b6171daebd7
I20260806 02:45:58.224431 13772 log.cc:1196] T c73a9404e32243139a524b6171daebd7 P 64a56e7a8ca6492680dd530522f4a3f9: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-2/wal/wals/c73a9404e32243139a524b6171daebd7
I20260806 02:45:58.224439 13774 ts_tablet_manager.cc:1918] T c73a9404e32243139a524b6171daebd7 P 0be05994eb164242b621cb037063e2d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:45:58.224524 13774 log.cc:1196] T c73a9404e32243139a524b6171daebd7 P 0be05994eb164242b621cb037063e2d0: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-1/wal/wals/c73a9404e32243139a524b6171daebd7
I20260806 02:45:58.224702 13772 ts_tablet_manager.cc:1939] T c73a9404e32243139a524b6171daebd7 P 64a56e7a8ca6492680dd530522f4a3f9: Deleting consensus metadata
I20260806 02:45:58.224702 13777 ts_tablet_manager.cc:1939] T c73a9404e32243139a524b6171daebd7 P 7d99f1b802814487b0455463222ed797: Deleting consensus metadata
I20260806 02:45:58.224795 13774 ts_tablet_manager.cc:1939] T c73a9404e32243139a524b6171daebd7 P 0be05994eb164242b621cb037063e2d0: Deleting consensus metadata
I20260806 02:45:58.225633 12983 catalog_manager.cc:4841] TS 7d99f1b802814487b0455463222ed797 (127.1.183.129:43601): tablet c73a9404e32243139a524b6171daebd7 (table test-table [id=bd6fa72f58d24b108e1a83f3648061ac]) successfully deleted
I20260806 02:45:58.225821 12983 catalog_manager.cc:4841] TS 64a56e7a8ca6492680dd530522f4a3f9 (127.1.183.131:33073): tablet c73a9404e32243139a524b6171daebd7 (table test-table [id=bd6fa72f58d24b108e1a83f3648061ac]) successfully deleted
I20260806 02:45:58.226038 12979 catalog_manager.cc:4841] TS 0be05994eb164242b621cb037063e2d0 (127.1.183.130:44967): tablet c73a9404e32243139a524b6171daebd7 (table test-table [id=bd6fa72f58d24b108e1a83f3648061ac]) successfully deleted
I20260806 02:45:58.228467 13201 tablet_service.cc:1467] Processing CreateTablet for tablet 18c573b913cb4ae5a14431290cfd3432 (DEFAULT_TABLE table=test-table [id=6cadc1e590044180b4a25115243236a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:58.228557 13404 tablet_service.cc:1467] Processing CreateTablet for tablet 18c573b913cb4ae5a14431290cfd3432 (DEFAULT_TABLE table=test-table [id=6cadc1e590044180b4a25115243236a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:58.228633 13599 tablet_service.cc:1467] Processing CreateTablet for tablet 18c573b913cb4ae5a14431290cfd3432 (DEFAULT_TABLE table=test-table [id=6cadc1e590044180b4a25115243236a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:45:58.228750 13201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18c573b913cb4ae5a14431290cfd3432. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:58.228766 13404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18c573b913cb4ae5a14431290cfd3432. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:58.228868 13599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18c573b913cb4ae5a14431290cfd3432. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:45:58.230682 13735 tablet_bootstrap.cc:492] T 18c573b913cb4ae5a14431290cfd3432 P 0be05994eb164242b621cb037063e2d0: Bootstrap starting.
I20260806 02:45:58.231133 13737 tablet_bootstrap.cc:492] T 18c573b913cb4ae5a14431290cfd3432 P 64a56e7a8ca6492680dd530522f4a3f9: Bootstrap starting.
I20260806 02:45:58.231133 13738 tablet_bootstrap.cc:492] T 18c573b913cb4ae5a14431290cfd3432 P 7d99f1b802814487b0455463222ed797: Bootstrap starting.
I20260806 02:45:58.231564 13735 tablet_bootstrap.cc:654] T 18c573b913cb4ae5a14431290cfd3432 P 0be05994eb164242b621cb037063e2d0: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:58.231997 13738 tablet_bootstrap.cc:654] T 18c573b913cb4ae5a14431290cfd3432 P 7d99f1b802814487b0455463222ed797: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:58.231997 13737 tablet_bootstrap.cc:654] T 18c573b913cb4ae5a14431290cfd3432 P 64a56e7a8ca6492680dd530522f4a3f9: Neither blocks nor log segments found. Creating new log.
I20260806 02:45:58.233148 13735 tablet_bootstrap.cc:492] T 18c573b913cb4ae5a14431290cfd3432 P 0be05994eb164242b621cb037063e2d0: No bootstrap required, opened a new log
I20260806 02:45:58.233197 13737 tablet_bootstrap.cc:492] T 18c573b913cb4ae5a14431290cfd3432 P 64a56e7a8ca6492680dd530522f4a3f9: No bootstrap required, opened a new log
I20260806 02:45:58.233247 13735 ts_tablet_manager.cc:1397] T 18c573b913cb4ae5a14431290cfd3432 P 0be05994eb164242b621cb037063e2d0: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260806 02:45:58.233268 13737 ts_tablet_manager.cc:1397] T 18c573b913cb4ae5a14431290cfd3432 P 64a56e7a8ca6492680dd530522f4a3f9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 02:45:58.233273 13738 tablet_bootstrap.cc:492] T 18c573b913cb4ae5a14431290cfd3432 P 7d99f1b802814487b0455463222ed797: No bootstrap required, opened a new log
I20260806 02:45:58.233376 13738 ts_tablet_manager.cc:1397] T 18c573b913cb4ae5a14431290cfd3432 P 7d99f1b802814487b0455463222ed797: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260806 02:45:58.233692 13737 raft_consensus.cc:348] T 18c573b913cb4ae5a14431290cfd3432 P 64a56e7a8ca6492680dd530522f4a3f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:58.233692 13735 raft_consensus.cc:348] T 18c573b913cb4ae5a14431290cfd3432 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:58.233815 13737 raft_consensus.cc:374] T 18c573b913cb4ae5a14431290cfd3432 P 64a56e7a8ca6492680dd530522f4a3f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:58.233815 13735 raft_consensus.cc:374] T 18c573b913cb4ae5a14431290cfd3432 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:58.233759 13738 raft_consensus.cc:348] T 18c573b913cb4ae5a14431290cfd3432 P 7d99f1b802814487b0455463222ed797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:58.233851 13735 raft_consensus.cc:729] T 18c573b913cb4ae5a14431290cfd3432 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0be05994eb164242b621cb037063e2d0, State: Initialized, Role: FOLLOWER
I20260806 02:45:58.233851 13737 raft_consensus.cc:729] T 18c573b913cb4ae5a14431290cfd3432 P 64a56e7a8ca6492680dd530522f4a3f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64a56e7a8ca6492680dd530522f4a3f9, State: Initialized, Role: FOLLOWER
I20260806 02:45:58.233896 13738 raft_consensus.cc:374] T 18c573b913cb4ae5a14431290cfd3432 P 7d99f1b802814487b0455463222ed797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:45:58.233930 13738 raft_consensus.cc:729] T 18c573b913cb4ae5a14431290cfd3432 P 7d99f1b802814487b0455463222ed797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d99f1b802814487b0455463222ed797, State: Initialized, Role: FOLLOWER
I20260806 02:45:58.233982 13735 consensus_queue.cc:260] T 18c573b913cb4ae5a14431290cfd3432 P 0be05994eb164242b621cb037063e2d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:58.233982 13737 consensus_queue.cc:260] T 18c573b913cb4ae5a14431290cfd3432 P 64a56e7a8ca6492680dd530522f4a3f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:58.234069 13738 consensus_queue.cc:260] T 18c573b913cb4ae5a14431290cfd3432 P 7d99f1b802814487b0455463222ed797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:58.234258 13735 ts_tablet_manager.cc:1428] T 18c573b913cb4ae5a14431290cfd3432 P 0be05994eb164242b621cb037063e2d0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:45:58.234258 13737 ts_tablet_manager.cc:1428] T 18c573b913cb4ae5a14431290cfd3432 P 64a56e7a8ca6492680dd530522f4a3f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:45:58.234381 13738 ts_tablet_manager.cc:1428] T 18c573b913cb4ae5a14431290cfd3432 P 7d99f1b802814487b0455463222ed797: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 02:45:58.278769 13810 raft_consensus.cc:482] T 18c573b913cb4ae5a14431290cfd3432 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:45:58.278962 13810 raft_consensus.cc:504] T 18c573b913cb4ae5a14431290cfd3432 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:58.279457 13810 leader_election.cc:288] T 18c573b913cb4ae5a14431290cfd3432 P 0be05994eb164242b621cb037063e2d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d99f1b802814487b0455463222ed797 (127.1.183.129:43601), 64a56e7a8ca6492680dd530522f4a3f9 (127.1.183.131:33073)
I20260806 02:45:58.280061 13627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c573b913cb4ae5a14431290cfd3432" candidate_uuid: "0be05994eb164242b621cb037063e2d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64a56e7a8ca6492680dd530522f4a3f9" is_pre_election: true
I20260806 02:45:58.280046 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c573b913cb4ae5a14431290cfd3432" candidate_uuid: "0be05994eb164242b621cb037063e2d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d99f1b802814487b0455463222ed797" is_pre_election: true
I20260806 02:45:58.280241 13627 raft_consensus.cc:2454] T 18c573b913cb4ae5a14431290cfd3432 P 64a56e7a8ca6492680dd530522f4a3f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0be05994eb164242b621cb037063e2d0 in term 0.
I20260806 02:45:58.280241 13234 raft_consensus.cc:2454] T 18c573b913cb4ae5a14431290cfd3432 P 7d99f1b802814487b0455463222ed797 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0be05994eb164242b621cb037063e2d0 in term 0.
I20260806 02:45:58.280616 13344 leader_election.cc:302] T 18c573b913cb4ae5a14431290cfd3432 P 0be05994eb164242b621cb037063e2d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0be05994eb164242b621cb037063e2d0, 7d99f1b802814487b0455463222ed797; no voters:
I20260806 02:45:58.280838 13810 raft_consensus.cc:2781] T 18c573b913cb4ae5a14431290cfd3432 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:45:58.280921 13810 raft_consensus.cc:482] T 18c573b913cb4ae5a14431290cfd3432 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:45:58.280961 13810 raft_consensus.cc:3037] T 18c573b913cb4ae5a14431290cfd3432 P 0be05994eb164242b621cb037063e2d0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:58.282317 13810 raft_consensus.cc:504] T 18c573b913cb4ae5a14431290cfd3432 P 0be05994eb164242b621cb037063e2d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:58.282723 13810 leader_election.cc:288] T 18c573b913cb4ae5a14431290cfd3432 P 0be05994eb164242b621cb037063e2d0 [CANDIDATE]: Term 1 election: Requested vote from peers 7d99f1b802814487b0455463222ed797 (127.1.183.129:43601), 64a56e7a8ca6492680dd530522f4a3f9 (127.1.183.131:33073)
I20260806 02:45:58.283172 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c573b913cb4ae5a14431290cfd3432" candidate_uuid: "0be05994eb164242b621cb037063e2d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d99f1b802814487b0455463222ed797"
I20260806 02:45:58.283315 13234 raft_consensus.cc:3037] T 18c573b913cb4ae5a14431290cfd3432 P 7d99f1b802814487b0455463222ed797 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:58.283313 13627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c573b913cb4ae5a14431290cfd3432" candidate_uuid: "0be05994eb164242b621cb037063e2d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64a56e7a8ca6492680dd530522f4a3f9"
I20260806 02:45:58.283434 13627 raft_consensus.cc:3037] T 18c573b913cb4ae5a14431290cfd3432 P 64a56e7a8ca6492680dd530522f4a3f9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:45:58.284611 13234 raft_consensus.cc:2454] T 18c573b913cb4ae5a14431290cfd3432 P 7d99f1b802814487b0455463222ed797 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0be05994eb164242b621cb037063e2d0 in term 1.
I20260806 02:45:58.284611 13627 raft_consensus.cc:2454] T 18c573b913cb4ae5a14431290cfd3432 P 64a56e7a8ca6492680dd530522f4a3f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0be05994eb164242b621cb037063e2d0 in term 1.
I20260806 02:45:58.284929 13344 leader_election.cc:302] T 18c573b913cb4ae5a14431290cfd3432 P 0be05994eb164242b621cb037063e2d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0be05994eb164242b621cb037063e2d0, 64a56e7a8ca6492680dd530522f4a3f9; no voters:
I20260806 02:45:58.285142 13810 raft_consensus.cc:2781] T 18c573b913cb4ae5a14431290cfd3432 P 0be05994eb164242b621cb037063e2d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:45:58.285408 13810 raft_consensus.cc:686] T 18c573b913cb4ae5a14431290cfd3432 P 0be05994eb164242b621cb037063e2d0 [term 1 LEADER]: Becoming Leader. State: Replica: 0be05994eb164242b621cb037063e2d0, State: Running, Role: LEADER
I20260806 02:45:58.285578 13810 consensus_queue.cc:237] T 18c573b913cb4ae5a14431290cfd3432 P 0be05994eb164242b621cb037063e2d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } }
I20260806 02:45:58.287412 13000 catalog_manager.cc:5495] T 18c573b913cb4ae5a14431290cfd3432 P 0be05994eb164242b621cb037063e2d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0be05994eb164242b621cb037063e2d0 (127.1.183.130). New cstate: current_term: 1 leader_uuid: "0be05994eb164242b621cb037063e2d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0be05994eb164242b621cb037063e2d0" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 44967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 } health_report { overall_health: UNKNOWN } } }
I20260806 02:45:58.312294 13855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:58.308981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43534 (local address 127.1.183.130:44967)
0806 02:45:58.309233 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:58.309237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:58.309249 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:45:58.309292 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:58.309294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:58.309339 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:58.309417 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:58.309686 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:58.310226 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:58.311116 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:58.311366 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:58.311484 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:58.311759 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:58.311849 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:58.311933 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:58.312034 (+ 101us) server_negotiation.cc:299] Negotiation successful
0806 02:45:58.312093 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":113,"threads_started":1}
I20260806 02:45:58.321928 13855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:58.318432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:38309 (local address 127.1.183.130:44967)
0806 02:45:58.318593 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 02:45:58.318597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:45:58.318766 (+ 169us) server_negotiation.cc:407] Connection header received
0806 02:45:58.318831 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:45:58.318834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:45:58.318877 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:45:58.318956 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:45:58.319196 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:58.319720 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:58.320787 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:45:58.321007 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:45:58.321114 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:58.321326 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:45:58.321400 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:45:58.321463 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 02:45:58.321555 (+ 92us) server_negotiation.cc:299] Negotiation successful
0806 02:45:58.321625 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":63}
I20260806 02:45:58.321929 13859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:58.318344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:44967 (local address 127.1.183.130:38309)
0806 02:45:58.318644 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:58.318664 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:45:58.318757 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:45:58.318980 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:45:58.318982 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:45:58.318988 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:45:58.319102 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:58.319106 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:58.319823 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:58.319826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:58.320620 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:45:58.320630 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:45:58.321132 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:45:58.321135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:45:58.321195 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:45:58.321222 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:45:58.321483 (+ 261us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:45:58.321488 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 02:45:58.321538 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 02:45:58.321656 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":15,"thread_start_us":129,"threads_started":1}
I20260806 02:45:58.329049 13627 raft_consensus.cc:1261] T 18c573b913cb4ae5a14431290cfd3432 P 64a56e7a8ca6492680dd530522f4a3f9 [term 1 FOLLOWER]: Refusing update from remote peer 0be05994eb164242b621cb037063e2d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:58.329075 13234 raft_consensus.cc:1261] T 18c573b913cb4ae5a14431290cfd3432 P 7d99f1b802814487b0455463222ed797 [term 1 FOLLOWER]: Refusing update from remote peer 0be05994eb164242b621cb037063e2d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:45:58.329473 13850 consensus_queue.cc:1035] T 18c573b913cb4ae5a14431290cfd3432 P 0be05994eb164242b621cb037063e2d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64a56e7a8ca6492680dd530522f4a3f9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 33073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:58.329639 13854 consensus_queue.cc:1035] T 18c573b913cb4ae5a14431290cfd3432 P 0be05994eb164242b621cb037063e2d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d99f1b802814487b0455463222ed797" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 43601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:45:58.364513 13000 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:58.364678 13000 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:45:58.366839 13000 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1557f14f7b246a3aee81db7b9d5a7eb: Sending DeleteTablet for 3 replicas of tablet 18c573b913cb4ae5a14431290cfd3432
I20260806 02:45:58.367570 13403 tablet_service.cc:1514] Processing DeleteTablet for tablet 18c573b913cb4ae5a14431290cfd3432 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:58 UTC) from {username='oryx', principal='oryx/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:43500
I20260806 02:45:58.367651 13201 tablet_service.cc:1514] Processing DeleteTablet for tablet 18c573b913cb4ae5a14431290cfd3432 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:58 UTC) from {username='oryx', principal='oryx/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:35030
I20260806 02:45:58.367681 13599 tablet_service.cc:1514] Processing DeleteTablet for tablet 18c573b913cb4ae5a14431290cfd3432 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:45:58 UTC) from {username='oryx', principal='oryx/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:37104
I20260806 02:45:58.367856 13777 tablet_replica.cc:330] T 18c573b913cb4ae5a14431290cfd3432 P 7d99f1b802814487b0455463222ed797: stopping tablet replica
I20260806 02:45:58.367872 13774 tablet_replica.cc:330] T 18c573b913cb4ae5a14431290cfd3432 P 0be05994eb164242b621cb037063e2d0: stopping tablet replica
I20260806 02:45:58.367990 13777 raft_consensus.cc:2229] T 18c573b913cb4ae5a14431290cfd3432 P 7d99f1b802814487b0455463222ed797 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:58.368013 13774 raft_consensus.cc:2229] T 18c573b913cb4ae5a14431290cfd3432 P 0be05994eb164242b621cb037063e2d0 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:45:58.368129 13777 raft_consensus.cc:2258] T 18c573b913cb4ae5a14431290cfd3432 P 7d99f1b802814487b0455463222ed797 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:58.368302 13772 tablet_replica.cc:330] T 18c573b913cb4ae5a14431290cfd3432 P 64a56e7a8ca6492680dd530522f4a3f9: stopping tablet replica
I20260806 02:45:58.368439 13772 raft_consensus.cc:2229] T 18c573b913cb4ae5a14431290cfd3432 P 64a56e7a8ca6492680dd530522f4a3f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:45:58.368577 13772 raft_consensus.cc:2258] T 18c573b913cb4ae5a14431290cfd3432 P 64a56e7a8ca6492680dd530522f4a3f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:58.368655 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 13108
I20260806 02:45:58.368783 13774 raft_consensus.cc:2258] T 18c573b913cb4ae5a14431290cfd3432 P 0be05994eb164242b621cb037063e2d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:45:58.368963 13777 ts_tablet_manager.cc:1905] T 18c573b913cb4ae5a14431290cfd3432 P 7d99f1b802814487b0455463222ed797: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:58.378167 13772 ts_tablet_manager.cc:1905] T 18c573b913cb4ae5a14431290cfd3432 P 64a56e7a8ca6492680dd530522f4a3f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:58.379765 13774 ts_tablet_manager.cc:1905] T 18c573b913cb4ae5a14431290cfd3432 P 0be05994eb164242b621cb037063e2d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:45:58.381582 13772 ts_tablet_manager.cc:1918] T 18c573b913cb4ae5a14431290cfd3432 P 64a56e7a8ca6492680dd530522f4a3f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:45:58.381690 13772 log.cc:1196] T 18c573b913cb4ae5a14431290cfd3432 P 64a56e7a8ca6492680dd530522f4a3f9: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-2/wal/wals/18c573b913cb4ae5a14431290cfd3432
I20260806 02:45:58.381986 13772 ts_tablet_manager.cc:1939] T 18c573b913cb4ae5a14431290cfd3432 P 64a56e7a8ca6492680dd530522f4a3f9: Deleting consensus metadata
I20260806 02:45:58.382097 13774 ts_tablet_manager.cc:1918] T 18c573b913cb4ae5a14431290cfd3432 P 0be05994eb164242b621cb037063e2d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:45:58.382206 13774 log.cc:1196] T 18c573b913cb4ae5a14431290cfd3432 P 0be05994eb164242b621cb037063e2d0: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984313887056-1758-0/minicluster-data/ts-1/wal/wals/18c573b913cb4ae5a14431290cfd3432
I20260806 02:45:58.382470 13774 ts_tablet_manager.cc:1939] T 18c573b913cb4ae5a14431290cfd3432 P 0be05994eb164242b621cb037063e2d0: Deleting consensus metadata
I20260806 02:45:58.383016 12983 catalog_manager.cc:4841] TS 64a56e7a8ca6492680dd530522f4a3f9 (127.1.183.131:33073): tablet 18c573b913cb4ae5a14431290cfd3432 (table test-table [id=6cadc1e590044180b4a25115243236a9]) successfully deleted
I20260806 02:45:58.383275 12979 catalog_manager.cc:4841] TS 0be05994eb164242b621cb037063e2d0 (127.1.183.130:44967): tablet 18c573b913cb4ae5a14431290cfd3432 (table test-table [id=6cadc1e590044180b4a25115243236a9]) successfully deleted
W20260806 02:45:58.387728 12983 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.183.129:43601 (error 108)
I20260806 02:45:58.388131 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 13315
I20260806 02:45:58.393656 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:45:58.391605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:43601 (local address 127.0.0.1:35054)
0806 02:45:58.393358 (+ 1753us) negotiation.cc:107] Waiting for socket to connect
0806 02:45:58.393491 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.129:43601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1657,"thread_start_us":1557,"threads_started":1}
W20260806 02:45:58.394886 12983 catalog_manager.cc:4584] TS 7d99f1b802814487b0455463222ed797 (127.1.183.129:43601): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 18c573b913cb4ae5a14431290cfd3432: Network error: Client connection negotiation failed: client connection to 127.1.183.129:43601: connect: Connection refused (error 111)
I20260806 02:45:58.406111 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 13506
I20260806 02:45:58.424121 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 12958
2026-08-06T02:45:58Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3747 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:45:58 dist-test-slave-gmjp krb5kdc[13874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:45:58 dist-test-slave-gmjp krb5kdc[13874](info): set up 2 sockets
Aug 06 02:45:58 dist-test-slave-gmjp krb5kdc[13874](info): commencing operation
krb5kdc: starting...
W20260806 02:45:59.678241 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.215s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:45:59 dist-test-slave-gmjp krb5kdc[13874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:45:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:45:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
I20260806 02:45:59.819669 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:36097
--webserver_interface=127.1.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:39319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:36097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:45:59.940317 13893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:45:59.940683 13893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:45:59.940779 13893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:45:59.945669 13893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:45:59.945780 13893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:45:59.945816 13893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:45:59.945842 13893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:45:59.945868 13893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:45:59.951732 13893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:39319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:36097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:36097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:45:59.953042 13893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:45:59.954155 13893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:45:59.963737 13899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:59.963833 13893 server_base.cc:1034] running on GCE node
W20260806 02:45:59.963793 13901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:45:59.963760 13898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:45:59.964720 13893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:45:59.966035 13893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:45:59.967244 13893 hybrid_clock.cc:648] HybridClock initialized: now 1785984359967215 us; error 44 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-06T02:45:59Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1538 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:46:00 dist-test-slave-gmjp krb5kdc[13906](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:46:00 dist-test-slave-gmjp krb5kdc[13906](info): set up 2 sockets
Aug 06 02:46:00 dist-test-slave-gmjp krb5kdc[13906](info): commencing operation
krb5kdc: starting...
W20260806 02:46:01.232139 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.230s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:46:01 dist-test-slave-gmjp krb5kdc[13906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:46:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:46:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
I20260806 02:46:01.376754 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:35427
--webserver_interface=127.1.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:35427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:46:01.503777 13929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:01.504098 13929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:01.504140 13929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:01.508343 13929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:46:01.508420 13929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:01.508440 13929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:01.508457 13929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:46:01.508473 13929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:46:01.514061 13929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:35427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:35427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:46:01.515254 13929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:01.516390 13929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:01.526422 13940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:01.526412 13936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:01.526465 13929 server_base.cc:1034] running on GCE node
W20260806 02:46:01.526412 13938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:01.527335 13929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:01.528579 13929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:01.529784 13929 hybrid_clock.cc:648] HybridClock initialized: now 1785984361529771 us; error 31 us; skew 500 ppm
W20260806 02:46:02.458689 13929 net_util.cc:504] Time spent look up canonical hostname for localhost 'dist-test-slave-gmjp': real 0.928s user 0.001s sys 0.000s
Aug 06 02:46:02 dist-test-slave-gmjp krb5kdc[13906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984362, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:02.461170 13929 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:46:02.462332 13929 webserver.cc:460] Webserver started at http://127.1.183.190:33249/ using document root <none> and password file <none>
I20260806 02:46:02.462991 13929 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:02.463090 13929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:02.463343 13929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:02.465631 13929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "6052c4cb2251491bbfdaf65d522e6126"
format_stamp: "Formatted at 2026-08-06 02:46:02 on dist-test-slave-gmjp"
server_key: "d50b439f8494a55c9850595af71ae28d"
server_key_iv: "1dd1838274ad4ce0f0c878ffb49fb32d"
server_key_version: "encryptionkey@0"
I20260806 02:46:02.466281 13929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "6052c4cb2251491bbfdaf65d522e6126"
format_stamp: "Formatted at 2026-08-06 02:46:02 on dist-test-slave-gmjp"
server_key: "d50b439f8494a55c9850595af71ae28d"
server_key_iv: "1dd1838274ad4ce0f0c878ffb49fb32d"
server_key_version: "encryptionkey@0"
I20260806 02:46:02.470522 13929 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:46:02.473519 13953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:02.475049 13929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:46:02.475221 13929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "6052c4cb2251491bbfdaf65d522e6126"
format_stamp: "Formatted at 2026-08-06 02:46:02 on dist-test-slave-gmjp"
server_key: "d50b439f8494a55c9850595af71ae28d"
server_key_iv: "1dd1838274ad4ce0f0c878ffb49fb32d"
server_key_version: "encryptionkey@0"
I20260806 02:46:02.475363 13929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:02.494719 13929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:02.498348 13929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:02.498591 13929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:02.506394 13929 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:35427
I20260806 02:46:02.506424 14020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:35427 every 8 connection(s)
I20260806 02:46:02.507467 13929 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:46:02.510476 14021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:02.512074 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 13929
I20260806 02:46:02.512291 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:46:02.512636 1758 external_mini_cluster.cc:1428] Setting key ff2169b5aebe8f76b27a7370dd30c8a7
I20260806 02:46:02.517167 14021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126: Bootstrap starting.
I20260806 02:46:02.519618 14021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:02.520584 14021 log.cc:824] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126: Log is configured to *not* fsync() on all Append() calls
Aug 06 02:46:02 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:46:02.522835 14021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126: No bootstrap required, opened a new log
I20260806 02:46:02.528045 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:02.514198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45472 (local address 127.1.183.190:35427)
0806 02:46:02.514673 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:02.514690 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:02.514728 (+ 38us) server_negotiation.cc:407] Connection header received
0806 02:46:02.515430 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:02.515448 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:02.515805 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:02.516164 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:02.517086 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:02.518140 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:02.519076 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:02.519580 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:02.519762 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:02.522468 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:02.522474 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:02.522489 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:02.522518 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:02.524916 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:02.525465 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:02.525471 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:02.525477 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:02.525609 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:02.525927 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:02.525931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:02.525933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:02.526378 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:02.526616 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:02.526899 (+ 283us) server_negotiation.cc:299] Negotiation successful
0806 02:46:02.527225 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":154,"threads_started":1}
I20260806 02:46:02.528102 14021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6052c4cb2251491bbfdaf65d522e6126" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35427 } }
I20260806 02:46:02.528447 14021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:02.528507 14021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6052c4cb2251491bbfdaf65d522e6126, State: Initialized, Role: FOLLOWER
I20260806 02:46:02.529023 14021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6052c4cb2251491bbfdaf65d522e6126" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35427 } }
I20260806 02:46:02.529187 14021 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:46:02.529241 14021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:46:02.529343 14021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:02.530375 14021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6052c4cb2251491bbfdaf65d522e6126" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35427 } }
I20260806 02:46:02.530778 14021 leader_election.cc:302] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6052c4cb2251491bbfdaf65d522e6126; no voters:
I20260806 02:46:02.531147 14021 leader_election.cc:288] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:46:02.531303 14028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:46:02.531576 14028 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [term 1 LEADER]: Becoming Leader. State: Replica: 6052c4cb2251491bbfdaf65d522e6126, State: Running, Role: LEADER
I20260806 02:46:02.531940 14028 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6052c4cb2251491bbfdaf65d522e6126" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35427 } }
I20260806 02:46:02.532583 14021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:46:02.533823 14031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6052c4cb2251491bbfdaf65d522e6126. Latest consensus state: current_term: 1 leader_uuid: "6052c4cb2251491bbfdaf65d522e6126" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6052c4cb2251491bbfdaf65d522e6126" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35427 } } }
I20260806 02:46:02.533962 14031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:02.534089 14030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6052c4cb2251491bbfdaf65d522e6126" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6052c4cb2251491bbfdaf65d522e6126" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35427 } } }
I20260806 02:46:02.534264 14030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [sys.catalog]: This master's current role is: LEADER
W20260806 02:46:02.538242 14047 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:46:02.538365 14047 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:46:02.538444 14039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:46:02.539412 14039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:46:02.545914 14039 catalog_manager.cc:1347] Generated new cluster ID: 457106405c55446eb58ca884e46cca20
I20260806 02:46:02.546017 14039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:46:02.558957 14039 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:46:02.560000 14039 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:46:02.577294 14039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126: Generated new TSK 0
I20260806 02:46:02.578192 14039 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
I20260806 02:46:02.620426 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.129:0
--local_ip_for_outbound_sockets=127.1.183.129
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:35427
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:46:02.743117 14056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:02.743422 14056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:02.743468 14056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:02.747690 14056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:02.747761 14056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:02.747823 14056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.129
I20260806 02:46:02.753358 14056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:35427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.129
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:46:02.754449 14056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:02.755630 14056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:02.765616 14065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:02.765595 14062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:02.765769 14056 server_base.cc:1034] running on GCE node
W20260806 02:46:02.765568 14063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:02.766343 14056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:02.767043 14056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:02.768239 14056 hybrid_clock.cc:648] HybridClock initialized: now 1785984362768220 us; error 30 us; skew 500 ppm
Aug 06 02:46:02 dist-test-slave-gmjp krb5kdc[13906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984362, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:02.771073 14056 init.cc:377] Logged in from keytab as kudu/127.1.183.129@KRBTEST.COM (short username kudu)
I20260806 02:46:02.772018 14056 webserver.cc:460] Webserver started at http://127.1.183.129:45243/ using document root <none> and password file <none>
I20260806 02:46:02.772707 14056 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:02.772811 14056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:02.773061 14056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:02.775231 14056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-0/data/instance:
uuid: "5069dc51540a478c9222d98184f56c63"
format_stamp: "Formatted at 2026-08-06 02:46:02 on dist-test-slave-gmjp"
server_key: "7ee0c50a70023c2194d91e4886de60d2"
server_key_iv: "76e96e1e066fdaf7eb14be3ed7422b89"
server_key_version: "encryptionkey@0"
I20260806 02:46:02.775854 14056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance:
uuid: "5069dc51540a478c9222d98184f56c63"
format_stamp: "Formatted at 2026-08-06 02:46:02 on dist-test-slave-gmjp"
server_key: "7ee0c50a70023c2194d91e4886de60d2"
server_key_iv: "76e96e1e066fdaf7eb14be3ed7422b89"
server_key_version: "encryptionkey@0"
I20260806 02:46:02.780015 14056 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 02:46:02.782835 14077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:02.784260 14056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:46:02.784420 14056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-0/wal
uuid: "5069dc51540a478c9222d98184f56c63"
format_stamp: "Formatted at 2026-08-06 02:46:02 on dist-test-slave-gmjp"
server_key: "7ee0c50a70023c2194d91e4886de60d2"
server_key_iv: "76e96e1e066fdaf7eb14be3ed7422b89"
server_key_version: "encryptionkey@0"
I20260806 02:46:02.784549 14056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:02.800089 14056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:02.803408 14056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:02.803633 14056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:02.804344 14056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:02.805226 14056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:02.805307 14056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:02.805387 14056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:02.805441 14056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:02.814997 14056 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.129:35121
I20260806 02:46:02.815021 14245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.129:35121 every 8 connection(s)
I20260806 02:46:02.815971 14056 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
I20260806 02:46:02.818610 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 14056
I20260806 02:46:02.818763 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance
I20260806 02:46:02.819082 1758 external_mini_cluster.cc:1428] Setting key 54caef205a28160bbef33462acf44af8
Aug 06 02:46:02 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984362, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.183.130@KRBTEST.COM; defaulting to no policy
I20260806 02:46:02.832666 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:02.817989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:35335 (local address 127.1.183.190:35427)
0806 02:46:02.818139 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:02.818143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:02.819066 (+ 923us) server_negotiation.cc:407] Connection header received
0806 02:46:02.820832 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:02.820835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:02.820900 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:02.820980 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:02.822268 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:02.823054 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:02.824375 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:02.824805 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:02.824943 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:02.828120 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:02.828127 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:02.828131 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:02.828193 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:02.829947 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:02.830557 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:02.830564 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:02.830567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:02.830669 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:02.831203 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:02.831207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:02.831211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:02.831447 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:02.831569 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:02.832311 (+ 742us) server_negotiation.cc:299] Negotiation successful
0806 02:46:02.832487 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:46:02.833796 14248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:02.818345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.129:35335)
0806 02:46:02.818894 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:02.818951 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:02.820576 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:02.821171 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:02.821179 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:02.821578 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:02.822102 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:02.822116 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:02.823214 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:02.823217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:02.824235 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:02.824247 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:02.824989 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:02.824992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:02.825156 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:02.825861 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:02.825879 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:02.827876 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:02.830111 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:02.830117 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:02.830134 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:02.830432 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:02.830772 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:02.830775 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:02.830776 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:02.831080 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:02.831569 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:02.831578 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:02.831931 (+ 353us) client_negotiation.cc:769] Sending connection context
0806 02:46:02.832259 (+ 328us) client_negotiation.cc:240] Negotiation successful
0806 02:46:02.832682 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":178,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.130@KRBTEST.COM" created.
I20260806 02:46:02.835316 14246 heartbeater.cc:347] Connected to a master server at 127.1.183.190:35427
I20260806 02:46:02.835615 14246 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:02.836231 14246 heartbeater.cc:511] Master 127.1.183.190:35427 requested a full tablet report, sending...
I20260806 02:46:02.838191 13973 ts_manager.cc:194] Registered new tserver with Master: 5069dc51540a478c9222d98184f56c63 (127.1.183.129:35121)
I20260806 02:46:02.839457 13973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.129@KRBTEST.COM'} at 127.1.183.129:35335
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
I20260806 02:46:02.852921 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.130:0
--local_ip_for_outbound_sockets=127.1.183.130
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:35427
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:46:02.977758 14253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:02.978142 14253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:02.978236 14253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:02.982693 14253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:02.982818 14253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:02.982929 14253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.130
I20260806 02:46:02.988710 14253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:35427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.130
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:46:02.990154 14253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:02.991528 14253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:03.002332 14258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:03.002422 14253 server_base.cc:1034] running on GCE node
W20260806 02:46:03.002332 14261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:03.002336 14264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:03.002977 14253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:03.003935 14253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:03.005133 14253 hybrid_clock.cc:648] HybridClock initialized: now 1785984363005106 us; error 47 us; skew 500 ppm
Aug 06 02:46:03 dist-test-slave-gmjp krb5kdc[13906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984363, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:03.008139 14253 init.cc:377] Logged in from keytab as kudu/127.1.183.130@KRBTEST.COM (short username kudu)
I20260806 02:46:03.009209 14253 webserver.cc:460] Webserver started at http://127.1.183.130:46785/ using document root <none> and password file <none>
I20260806 02:46:03.009784 14253 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:03.009850 14253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:03.010028 14253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:03.012351 14253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-1/data/instance:
uuid: "3f226ecc0aa04fb98adcacb14b41e7cb"
format_stamp: "Formatted at 2026-08-06 02:46:03 on dist-test-slave-gmjp"
server_key: "3d2203626ea1b8c323c90a90d9520661"
server_key_iv: "6b6495af10f5b3fe21a965227ccf98ed"
server_key_version: "encryptionkey@0"
I20260806 02:46:03.012933 14253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance:
uuid: "3f226ecc0aa04fb98adcacb14b41e7cb"
format_stamp: "Formatted at 2026-08-06 02:46:03 on dist-test-slave-gmjp"
server_key: "3d2203626ea1b8c323c90a90d9520661"
server_key_iv: "6b6495af10f5b3fe21a965227ccf98ed"
server_key_version: "encryptionkey@0"
I20260806 02:46:03.017068 14253 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 02:46:03.019778 14274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:03.021270 14253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:46:03.021426 14253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-1/wal
uuid: "3f226ecc0aa04fb98adcacb14b41e7cb"
format_stamp: "Formatted at 2026-08-06 02:46:03 on dist-test-slave-gmjp"
server_key: "3d2203626ea1b8c323c90a90d9520661"
server_key_iv: "6b6495af10f5b3fe21a965227ccf98ed"
server_key_version: "encryptionkey@0"
I20260806 02:46:03.021588 14253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:03.047930 14253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:03.051218 14253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:03.051403 14253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:03.052083 14253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:03.053030 14253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:03.053083 14253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:03.053125 14253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:03.053141 14253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:03.063577 14253 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.130:36145
I20260806 02:46:03.063604 14432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.130:36145 every 8 connection(s)
I20260806 02:46:03.064597 14253 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
I20260806 02:46:03.072232 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 14253
I20260806 02:46:03.072361 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance
I20260806 02:46:03.072682 1758 external_mini_cluster.cc:1428] Setting key 17082948448b92e909e320baf3782c4b
Aug 06 02:46:03 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984363, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:46:03.080339 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:03.066512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:41039 (local address 127.1.183.190:35427)
0806 02:46:03.066704 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:03.066709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:03.067736 (+ 1027us) server_negotiation.cc:407] Connection header received
0806 02:46:03.069110 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:03.069113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:03.069165 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:03.069251 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:03.070543 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.071102 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.072566 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.072841 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.072983 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:03.075704 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:03.075708 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:03.075711 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:03.075746 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:03.077712 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:03.078300 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:03.078307 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:03.078312 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:03.078422 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:03.078772 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:03.078779 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:03.078783 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:03.079087 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:03.079225 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:03.079882 (+ 657us) server_negotiation.cc:299] Negotiation successful
0806 02:46:03.080048 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 02:46:03.081247 14436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:03.067016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.130:41039)
0806 02:46:03.067548 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:03.067597 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:03.068872 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:03.069458 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:03.069464 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:03.069864 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:03.070380 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:03.070395 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.071254 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.071258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:03.072345 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:03.072361 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.073040 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.073044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:03.073260 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:03.073909 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:03.073931 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:03.075520 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:03.077851 (+ 2331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:03.077859 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:03.077878 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:03.078167 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:03.078539 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:03.078542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:03.078543 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:03.078690 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:03.079232 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:03.079238 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:03.079588 (+ 350us) client_negotiation.cc:769] Sending connection context
0806 02:46:03.079766 (+ 178us) client_negotiation.cc:240] Negotiation successful
0806 02:46:03.080225 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":123,"threads_started":1}
I20260806 02:46:03.082587 14433 heartbeater.cc:347] Connected to a master server at 127.1.183.190:35427
I20260806 02:46:03.082896 14433 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:03.083603 14433 heartbeater.cc:511] Master 127.1.183.190:35427 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.183.131@KRBTEST.COM; defaulting to no policy
I20260806 02:46:03.084942 13973 ts_manager.cc:194] Registered new tserver with Master: 3f226ecc0aa04fb98adcacb14b41e7cb (127.1.183.130:36145)
I20260806 02:46:03.085554 13973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.130@KRBTEST.COM'} at 127.1.183.130:41039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
I20260806 02:46:03.105250 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.131:0
--local_ip_for_outbound_sockets=127.1.183.131
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:35427
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:46:03.231114 14439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:03.231417 14439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:03.231464 14439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:03.236070 14439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:03.236166 14439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:03.236270 14439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.131
I20260806 02:46:03.241902 14439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:35427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.131
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:46:03.243111 14439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:03.244269 14439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:03.254437 14448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:03.254467 14447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:03.254645 14439 server_base.cc:1034] running on GCE node
W20260806 02:46:03.254484 14452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:03.255330 14439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:03.256023 14439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:03.257203 14439 hybrid_clock.cc:648] HybridClock initialized: now 1785984363257182 us; error 41 us; skew 500 ppm
Aug 06 02:46:03 dist-test-slave-gmjp krb5kdc[13906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984363, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:03.260236 14439 init.cc:377] Logged in from keytab as kudu/127.1.183.131@KRBTEST.COM (short username kudu)
I20260806 02:46:03.261420 14439 webserver.cc:460] Webserver started at http://127.1.183.131:35181/ using document root <none> and password file <none>
I20260806 02:46:03.262076 14439 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:03.262182 14439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:03.262435 14439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:03.264725 14439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-2/data/instance:
uuid: "a4d7b9cad7204d26a924ac32f1442e7f"
format_stamp: "Formatted at 2026-08-06 02:46:03 on dist-test-slave-gmjp"
server_key: "3d3c1876c37695ae76fda3ea390fce47"
server_key_iv: "654b99b2d1d26a1be3e0b9cba8b40b85"
server_key_version: "encryptionkey@0"
I20260806 02:46:03.265343 14439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance:
uuid: "a4d7b9cad7204d26a924ac32f1442e7f"
format_stamp: "Formatted at 2026-08-06 02:46:03 on dist-test-slave-gmjp"
server_key: "3d3c1876c37695ae76fda3ea390fce47"
server_key_iv: "654b99b2d1d26a1be3e0b9cba8b40b85"
server_key_version: "encryptionkey@0"
I20260806 02:46:03.269469 14439 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260806 02:46:03.272101 14463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:03.273504 14439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:46:03.273656 14439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-2/wal
uuid: "a4d7b9cad7204d26a924ac32f1442e7f"
format_stamp: "Formatted at 2026-08-06 02:46:03 on dist-test-slave-gmjp"
server_key: "3d3c1876c37695ae76fda3ea390fce47"
server_key_iv: "654b99b2d1d26a1be3e0b9cba8b40b85"
server_key_version: "encryptionkey@0"
I20260806 02:46:03.273782 14439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:03.292686 14439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:03.296247 14439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:03.296483 14439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:03.297165 14439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:03.298097 14439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:03.298170 14439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:03.298245 14439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:03.298288 14439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:03.307802 14439 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.131:38761
I20260806 02:46:03.307830 14625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.131:38761 every 8 connection(s)
I20260806 02:46:03.308799 14439 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
I20260806 02:46:03.314015 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 14439
I20260806 02:46:03.314188 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance
I20260806 02:46:03.314487 1758 external_mini_cluster.cc:1428] Setting key 1716325ce95cbf845cd789c01325e46d
Aug 06 02:46:03 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984363, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:46:03.323654 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:03.311006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:45259 (local address 127.1.183.190:35427)
0806 02:46:03.311210 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:03.311215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:03.312181 (+ 966us) server_negotiation.cc:407] Connection header received
0806 02:46:03.313561 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:03.313564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:03.313615 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:03.313728 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:03.315236 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.316053 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.317603 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.317819 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.317910 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:03.320516 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:03.320519 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:03.320521 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:03.320546 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:03.321812 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:03.322298 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:03.322300 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:03.322302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:03.322378 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:03.322677 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:03.322679 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:03.322680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:03.322858 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:03.322952 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:03.323422 (+ 470us) server_negotiation.cc:299] Negotiation successful
0806 02:46:03.323525 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 02:46:03.324646 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:03.311436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.131:45259)
0806 02:46:03.311978 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:03.312025 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:03.313333 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:03.313907 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:03.313914 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:03.314406 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:03.315091 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:03.315109 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.316232 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.316235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:03.317487 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:03.317498 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.317958 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.317961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:03.318102 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:03.318784 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:03.318801 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:03.320358 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:03.321920 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:03.321925 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:03.321939 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:03.322199 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:03.322456 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:03.322459 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:03.322460 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:03.322590 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:03.322961 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:03.322967 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:03.323202 (+ 235us) client_negotiation.cc:769] Sending connection context
0806 02:46:03.323362 (+ 160us) client_negotiation.cc:240] Negotiation successful
0806 02:46:03.323632 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":137,"threads_started":1}
I20260806 02:46:03.326140 14626 heartbeater.cc:347] Connected to a master server at 127.1.183.190:35427
I20260806 02:46:03.326383 14626 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:03.326885 14626 heartbeater.cc:511] Master 127.1.183.190:35427 requested a full tablet report, sending...
I20260806 02:46:03.327986 13972 ts_manager.cc:194] Registered new tserver with Master: a4d7b9cad7204d26a924ac32f1442e7f (127.1.183.131:38761)
I20260806 02:46:03.328670 13972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.131@KRBTEST.COM'} at 127.1.183.131:45259
I20260806 02:46:03.328809 1758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.129.keytab.
Entry for principal oryx/127.1.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.129.keytab.
WARNING: no policy specified for oryx/127.1.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.130.keytab.
Entry for principal oryx/127.1.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.130.keytab.
WARNING: no policy specified for oryx/127.1.183.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.183.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.131.keytab.
Entry for principal oryx/127.1.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.131.keytab.
I20260806 02:46:03.415552 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 14056
I20260806 02:46:03.432546 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 14253
I20260806 02:46:03.444242 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 14439
I20260806 02:46:03.466435 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 13929
I20260806 02:46:03.486341 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:35427
--webserver_interface=127.1.183.190
--webserver_port=33249
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:35427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:46:03.607583 14639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:03.607973 14639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:03.608065 14639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:03.612298 14639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:46:03.612418 14639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:03.612475 14639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:03.612541 14639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:46:03.612583 14639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:46:03.618232 14639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:35427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:35427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=33249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:46:03.619525 14639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:03.620697 14639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:03.629715 14646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:03.629715 14650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:03.629739 14647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:03.630026 14639 server_base.cc:1034] running on GCE node
I20260806 02:46:03.630652 14639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:03.631939 14639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:03.633149 14639 hybrid_clock.cc:648] HybridClock initialized: now 1785984363633158 us; error 54 us; skew 500 ppm
Aug 06 02:46:03 dist-test-slave-gmjp krb5kdc[13906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984363, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:03.636092 14639 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:46:03.637163 14639 webserver.cc:460] Webserver started at http://127.1.183.190:33249/ using document root <none> and password file <none>
I20260806 02:46:03.637792 14639 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:03.637916 14639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:03.642362 14639 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260806 02:46:03.644726 14659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:03.646158 14639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:46:03.646318 14639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "6052c4cb2251491bbfdaf65d522e6126"
format_stamp: "Formatted at 2026-08-06 02:46:02 on dist-test-slave-gmjp"
server_key: "d50b439f8494a55c9850595af71ae28d"
server_key_iv: "1dd1838274ad4ce0f0c878ffb49fb32d"
server_key_version: "encryptionkey@0"
I20260806 02:46:03.646916 14639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:03.660949 14639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:03.664568 14639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:03.664795 14639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:03.672394 14639 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:35427
I20260806 02:46:03.672408 14748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:35427 every 8 connection(s)
I20260806 02:46:03.673580 14639 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:46:03.674094 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 14639
I20260806 02:46:03.674575 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.129:35121
--local_ip_for_outbound_sockets=127.1.183.129
--tserver_master_addrs=127.1.183.190:35427
--webserver_port=45243
--webserver_interface=127.1.183.129
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.183.129
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.129.keytab with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:46:03.679085 14749 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:46:03.681038 14749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126: Bootstrap starting.
I20260806 02:46:03.699409 14749 log.cc:824] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126: Log is configured to *not* fsync() on all Append() calls
I20260806 02:46:03.706732 14749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:46:03.707185 14749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126: Bootstrap complete.
I20260806 02:46:03.711867 14749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6052c4cb2251491bbfdaf65d522e6126" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35427 } }
I20260806 02:46:03.712410 14749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6052c4cb2251491bbfdaf65d522e6126, State: Initialized, Role: FOLLOWER
I20260806 02:46:03.713006 14749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6052c4cb2251491bbfdaf65d522e6126" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35427 } }
I20260806 02:46:03.713165 14749 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:46:03.713260 14749 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:46:03.713389 14749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:46:03.715507 14749 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6052c4cb2251491bbfdaf65d522e6126" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35427 } }
I20260806 02:46:03.715873 14749 leader_election.cc:302] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6052c4cb2251491bbfdaf65d522e6126; no voters:
I20260806 02:46:03.716308 14749 leader_election.cc:288] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 02:46:03.716411 14760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 02:46:03.716616 14760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [term 2 LEADER]: Becoming Leader. State: Replica: 6052c4cb2251491bbfdaf65d522e6126, State: Running, Role: LEADER
I20260806 02:46:03.716929 14760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6052c4cb2251491bbfdaf65d522e6126" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35427 } }
I20260806 02:46:03.717675 14749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:46:03.718391 14764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6052c4cb2251491bbfdaf65d522e6126" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6052c4cb2251491bbfdaf65d522e6126" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35427 } } }
I20260806 02:46:03.718537 14764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:03.719507 14772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:46:03.719527 14765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6052c4cb2251491bbfdaf65d522e6126. Latest consensus state: current_term: 2 leader_uuid: "6052c4cb2251491bbfdaf65d522e6126" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6052c4cb2251491bbfdaf65d522e6126" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35427 } } }
I20260806 02:46:03.719784 14765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126 [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:03.723143 14772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:46:03.723815 14772 catalog_manager.cc:1259] Loaded cluster ID: 457106405c55446eb58ca884e46cca20
I20260806 02:46:03.723881 14772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:46:03.726364 14772 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:46:03.727077 14772 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6052c4cb2251491bbfdaf65d522e6126: Loaded TSK: 0
I20260806 02:46:03.728390 14772 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:46:03.797992 14757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:03.798355 14757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:03.798437 14757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:03.802927 14757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:03.803030 14757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:03.803128 14757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.129
I20260806 02:46:03.808768 14757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.129.keytab
--principal=oryx/127.1.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.129:35121
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.183.129
--webserver_port=45243
--tserver_master_addrs=127.1.183.190:35427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.129
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:46:03.809911 14757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:03.811002 14757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:03.822440 14799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:03.822444 14803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:03.822837 14801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:03.823413 14757 server_base.cc:1034] running on GCE node
I20260806 02:46:03.823977 14757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:03.824962 14757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:03.826179 14757 hybrid_clock.cc:648] HybridClock initialized: now 1785984363826158 us; error 34 us; skew 500 ppm
Aug 06 02:46:03 dist-test-slave-gmjp krb5kdc[13906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984363, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:03.830032 14757 init.cc:377] Logged in from keytab as oryx/127.1.183.129@KRBTEST.COM (short username oryx)
I20260806 02:46:03.831374 14757 webserver.cc:460] Webserver started at http://127.1.183.129:45243/ using document root <none> and password file <none>
I20260806 02:46:03.832213 14757 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:03.832345 14757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:03.838119 14757 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:46:03.841269 14814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:03.843060 14757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:46:03.843227 14757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-0/wal
uuid: "5069dc51540a478c9222d98184f56c63"
format_stamp: "Formatted at 2026-08-06 02:46:02 on dist-test-slave-gmjp"
server_key: "7ee0c50a70023c2194d91e4886de60d2"
server_key_iv: "76e96e1e066fdaf7eb14be3ed7422b89"
server_key_version: "encryptionkey@0"
I20260806 02:46:03.844051 14757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:03.864619 14757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:03.869016 14757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:03.869241 14757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:03.870067 14757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:03.871371 14757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:03.871457 14757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:03.871517 14757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:03.871551 14757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:03.883124 14757 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.129:35121
I20260806 02:46:03.883215 14986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.129:35121 every 8 connection(s)
I20260806 02:46:03.884645 14757 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
I20260806 02:46:03.893532 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 14757
I20260806 02:46:03.894102 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.130:36145
--local_ip_for_outbound_sockets=127.1.183.130
--tserver_master_addrs=127.1.183.190:35427
--webserver_port=46785
--webserver_interface=127.1.183.130
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.183.130
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.130.keytab with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:46:03 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:03 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 02:46:03.900203 14990 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:03.887154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:48995 (local address 127.1.183.190:35427)
0806 02:46:03.887711 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:03.887733 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:03.888654 (+ 921us) server_negotiation.cc:407] Connection header received
0806 02:46:03.890299 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:03.890312 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:03.890682 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:03.891050 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:03.892350 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.893355 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.894760 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.895267 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.895479 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:03.899336 (+ 3857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.129:48995: BlockingRecv error: recv got EOF from 127.1.183.129:48995 (error 108)
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":146,"threads_started":1}
I20260806 02:46:03.900207 14991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:03.887233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.129:48995)
0806 02:46:03.888462 (+ 1229us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:03.888509 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:03.889801 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:03.891149 (+ 1348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:03.891158 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:03.891612 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:03.892110 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:03.892125 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.893523 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.893527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:03.894623 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:03.894634 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.895463 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.895468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:03.895665 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:03.896364 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:03.896383 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:03.899096 (+ 2713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1036,"thread_start_us":156,"threads_started":1}
W20260806 02:46:03.901059 14988 heartbeater.cc:644] Failed to heartbeat to 127.1.183.190:35427 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.183.190:35427: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Aug 06 02:46:03 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:03 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:03.907270 14991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:03.901654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.129:54601)
0806 02:46:03.901806 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:03.901821 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:03.901918 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:03.902197 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:03.902200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:03.902380 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:03.902487 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:03.902492 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.903253 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.903258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:03.903816 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:03.903825 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.904361 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.904365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:03.904473 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:03.905048 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:03.905060 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:03.907109 (+ 2049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 02:46:03.907385 14990 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:03.901747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:54601 (local address 127.1.183.190:35427)
0806 02:46:03.901870 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:03.901875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:03.901889 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:46:03.902045 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:03.902048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:03.902094 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:03.902179 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:03.902604 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.903136 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.903956 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.904243 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.904355 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:03.907250 (+ 2895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.129:54601: BlockingRecv error: recv got EOF from 127.1.183.129:54601 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 06 02:46:03 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:03 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:03.913753 14991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:03.907991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.129:46523)
0806 02:46:03.908180 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:03.908195 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:03.908269 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:03.908501 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:03.908504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:03.908675 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:03.908788 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:03.908794 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.909491 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.909495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:03.910383 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:03.910394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.910841 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.910845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:03.910932 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:03.911421 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:03.911436 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:03.913583 (+ 2147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104}
W20260806 02:46:03.913996 14988 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:46:03.913851 14990 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:03.908077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:46523 (local address 127.1.183.190:35427)
0806 02:46:03.908230 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:03.908233 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:03.908246 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:46:03.908351 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:03.908353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:03.908391 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:03.908466 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:03.908900 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.909385 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.910517 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:03.910731 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:03.910808 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:03.913717 (+ 2909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.129:46523: BlockingRecv error: recv got EOF from 127.1.183.129:46523 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260806 02:46:04.019722 14992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:04.020077 14992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:04.020160 14992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:04.024570 14992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:04.024642 14992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:04.024739 14992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.130
I20260806 02:46:04.030354 14992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.130.keytab
--principal=oryx/127.1.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.130:36145
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.183.130
--webserver_port=46785
--tserver_master_addrs=127.1.183.190:35427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.130
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:46:04.031607 14992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:04.032823 14992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:04.043272 15001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:04.043318 14999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:04.043332 15004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:04.043764 14992 server_base.cc:1034] running on GCE node
I20260806 02:46:04.044332 14992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:04.045105 14992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:04.046309 14992 hybrid_clock.cc:648] HybridClock initialized: now 1785984364046296 us; error 33 us; skew 500 ppm
Aug 06 02:46:04 dist-test-slave-gmjp krb5kdc[13906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984364, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:04.049574 14992 init.cc:377] Logged in from keytab as oryx/127.1.183.130@KRBTEST.COM (short username oryx)
I20260806 02:46:04.050802 14992 webserver.cc:460] Webserver started at http://127.1.183.130:46785/ using document root <none> and password file <none>
I20260806 02:46:04.051443 14992 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:04.051527 14992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:04.055876 14992 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:46:04.058485 15013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:04.059892 14992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 02:46:04.060014 14992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-1/wal
uuid: "3f226ecc0aa04fb98adcacb14b41e7cb"
format_stamp: "Formatted at 2026-08-06 02:46:03 on dist-test-slave-gmjp"
server_key: "3d2203626ea1b8c323c90a90d9520661"
server_key_iv: "6b6495af10f5b3fe21a965227ccf98ed"
server_key_version: "encryptionkey@0"
I20260806 02:46:04.060668 14992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:04.072003 14992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:04.075376 14992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:04.075554 14992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:04.076176 14992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:04.077104 14992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:04.077193 14992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:04.077291 14992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:04.077340 14992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:04.087203 14992 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.130:36145
I20260806 02:46:04.087220 15174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.130:36145 every 8 connection(s)
I20260806 02:46:04.088342 14992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
I20260806 02:46:04.092826 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 14992
I20260806 02:46:04.093319 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.131:38761
--local_ip_for_outbound_sockets=127.1.183.131
--tserver_master_addrs=127.1.183.190:35427
--webserver_port=35181
--webserver_interface=127.1.183.131
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.183.131
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.131.keytab with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:46:04 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:04 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 02:46:04.101188 14990 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:04.090248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:47069 (local address 127.1.183.190:35427)
0806 02:46:04.090407 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:04.090411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:04.091225 (+ 814us) server_negotiation.cc:407] Connection header received
0806 02:46:04.092683 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:04.092687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:04.092756 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:04.092873 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:04.094211 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.094753 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.095951 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.096249 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.096378 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.101001 (+ 4623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.130:47069: BlockingRecv error: recv got EOF from 127.1.183.130:47069 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:46:04.101814 15177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:04.090587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.130:47069)
0806 02:46:04.091060 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:04.091104 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:04.092441 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:04.093066 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:04.093073 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:04.093501 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:04.094039 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:04.094054 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.094900 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.094906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:04.095807 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:04.095817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.096459 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.096464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:04.096712 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.097454 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:04.097482 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:04.100765 (+ 3283us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":121,"threads_started":1}
W20260806 02:46:04.102514 15175 heartbeater.cc:644] Failed to heartbeat to 127.1.183.190:35427 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.183.190:35427: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Aug 06 02:46:04 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:04 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:04.108558 15177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:04.103029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.130:58237)
0806 02:46:04.103155 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:04.103172 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:04.103264 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:04.103549 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:04.103552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:04.103735 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:04.103844 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:04.103849 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.104688 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.104693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:04.105256 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:04.105264 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.105727 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.105731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:04.105833 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.106399 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:04.106411 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:04.108388 (+ 1977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 02:46:04.108661 14990 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:04.103173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:58237 (local address 127.1.183.190:35427)
0806 02:46:04.103329 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:04.103335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:04.103352 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:46:04.103402 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:04.103404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:04.103442 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:04.103530 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:04.103972 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.104574 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.105397 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.105609 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.105720 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.108534 (+ 2814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.130:58237: BlockingRecv error: recv got EOF from 127.1.183.130:58237 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 06 02:46:04 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:04 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:04.115139 15177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:04.109327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.130:39385)
0806 02:46:04.109491 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:04.109506 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:04.109591 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:04.109826 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:04.109829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:04.110002 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:04.110099 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:04.110106 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.110834 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.110837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:04.111719 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:04.111729 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.112215 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.112222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:04.112331 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.112833 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:04.112847 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:04.114981 (+ 2134us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":77}
W20260806 02:46:04.115312 14990 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:04.109459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:39385 (local address 127.1.183.190:35427)
0806 02:46:04.109569 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:04.109573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:04.109586 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:46:04.109680 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:04.109682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:04.109720 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:04.109787 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:04.110221 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.110728 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.111844 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.112065 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.112146 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.115146 (+ 3000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.130:39385: BlockingRecv error: recv got EOF from 127.1.183.130:39385 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260806 02:46:04.115617 15175 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:46:04.218124 15179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:04.218591 15179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:04.218675 15179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:04.223086 15179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:04.223194 15179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:04.223294 15179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.131
I20260806 02:46:04.229355 15179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/krb5kdc/oryx_127.1.183.131.keytab
--principal=oryx/127.1.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.131:38761
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.183.131
--webserver_port=35181
--tserver_master_addrs=127.1.183.190:35427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.131
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:46:04.230533 15179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:04.231640 15179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:04.242028 15187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:04.242027 15184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:04.242022 15185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:04.242339 15179 server_base.cc:1034] running on GCE node
I20260806 02:46:04.242901 15179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:04.243628 15179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:04.244815 15179 hybrid_clock.cc:648] HybridClock initialized: now 1785984364244799 us; error 32 us; skew 500 ppm
Aug 06 02:46:04 dist-test-slave-gmjp krb5kdc[13906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984364, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:04.247654 15179 init.cc:377] Logged in from keytab as oryx/127.1.183.131@KRBTEST.COM (short username oryx)
I20260806 02:46:04.248797 15179 webserver.cc:460] Webserver started at http://127.1.183.131:35181/ using document root <none> and password file <none>
I20260806 02:46:04.249429 15179 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:04.249554 15179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:04.254191 15179 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:46:04.256654 15200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:04.258101 15179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:46:04.258279 15179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-2/wal
uuid: "a4d7b9cad7204d26a924ac32f1442e7f"
format_stamp: "Formatted at 2026-08-06 02:46:03 on dist-test-slave-gmjp"
server_key: "3d3c1876c37695ae76fda3ea390fce47"
server_key_iv: "654b99b2d1d26a1be3e0b9cba8b40b85"
server_key_version: "encryptionkey@0"
I20260806 02:46:04.259058 15179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:04.281920 15179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:04.285266 15179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:04.285683 15179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:04.286427 15179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:04.287393 15179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:04.287472 15179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:04.287549 15179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:04.287592 15179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:04.298282 15179 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.131:38761
I20260806 02:46:04.298312 15367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.131:38761 every 8 connection(s)
I20260806 02:46:04.299454 15179 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
I20260806 02:46:04.302847 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 15179
Aug 06 02:46:04 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:04 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 02:46:04.313623 14990 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:04.301483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:40807 (local address 127.1.183.190:35427)
0806 02:46:04.301646 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:04.301650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:04.302697 (+ 1047us) server_negotiation.cc:407] Connection header received
0806 02:46:04.304616 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:04.304622 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:04.304689 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:04.304866 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:04.306452 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.307107 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.308893 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.309260 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.309354 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.313480 (+ 4126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.131:40807: BlockingRecv error: recv got EOF from 127.1.183.131:40807 (error 108)
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":64,"spinlock_wait_cycles":2688}
I20260806 02:46:04.314360 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:04.301902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.131:40807)
0806 02:46:04.302460 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:04.302543 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:04.304026 (+ 1483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:04.305094 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:04.305104 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:04.305694 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:04.306280 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:04.306304 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.307249 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.307252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:04.308735 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:04.308748 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.309584 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.309588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:04.309842 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.310548 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:04.310565 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:04.313364 (+ 2799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":128,"threads_started":1}
W20260806 02:46:04.315058 15368 heartbeater.cc:644] Failed to heartbeat to 127.1.183.190:35427 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.183.190:35427: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
I20260806 02:46:04.315418 15375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:04.304256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45488 (local address 127.1.183.190:35427)
0806 02:46:04.304589 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:04.304595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:04.304610 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:46:04.304644 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:04.304647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:04.304762 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:04.304866 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:04.305909 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.306622 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.307732 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.308354 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.308484 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.310435 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:04.310442 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:04.310459 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:04.310488 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:04.312556 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:04.313067 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:04.313074 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:04.313079 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:04.313216 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:04.313532 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:04.313537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:04.313538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:04.314025 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:04.314260 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:04.315022 (+ 762us) server_negotiation.cc:299] Negotiation successful
0806 02:46:04.315289 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":213,"spinlock_wait_cycles":14976,"thread_start_us":104,"threads_started":1}
Aug 06 02:46:04 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:04 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:04.321708 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:04.315579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.131:53379)
0806 02:46:04.315768 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:04.315792 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:04.315873 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:04.316102 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:04.316105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:04.316302 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:04.316430 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:04.316438 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.317483 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.317489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:04.318243 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:04.318256 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.319002 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.319005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:04.319104 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.319638 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:04.319652 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:04.321546 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113}
W20260806 02:46:04.321854 15375 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:04.315723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:53379 (local address 127.1.183.190:35427)
0806 02:46:04.315841 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:04.315844 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:04.315857 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:46:04.315955 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:04.315957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:04.315993 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:04.316058 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:04.316548 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.317345 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.318366 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.318788 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.319001 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.321688 (+ 2687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.131:53379: BlockingRecv error: recv got EOF from 127.1.183.131:53379 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 06 02:46:04 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:04 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:04.327167 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:04.322409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.131:58633)
0806 02:46:04.322537 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:04.322554 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:04.322634 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:04.322868 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:04.322870 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:04.323005 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:04.323082 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:04.323086 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.323768 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.323770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:04.324373 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:04.324382 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.324798 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.324801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:04.324859 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.325296 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:04.325306 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:04.327058 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 02:46:04.327355 15375 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:04.322490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:58633 (local address 127.1.183.190:35427)
0806 02:46:04.322671 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:04.322675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:04.322687 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:46:04.322722 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:04.322724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:04.322759 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:04.322839 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:04.323186 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.323675 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.324488 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.324698 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.324812 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.327215 (+ 2403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.131:58633: BlockingRecv error: recv got EOF from 127.1.183.131:58633 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260806 02:46:04.327476 15368 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 06 02:46:04 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:04 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:04.920465 15376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:04.914710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.129:48383)
0806 02:46:04.915101 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:04.915118 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:04.915241 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:04.915503 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:04.915505 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:04.915687 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:04.915789 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:04.915794 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.916561 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.916563 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:04.917135 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:04.917144 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.917601 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.917603 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:04.917668 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.918222 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:04.918236 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:04.920325 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":104,"threads_started":1}
W20260806 02:46:04.920687 15378 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:04.914769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:48383 (local address 127.1.183.190:35427)
0806 02:46:04.915091 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:04.915096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:04.915250 (+ 154us) server_negotiation.cc:407] Connection header received
0806 02:46:04.915335 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:04.915338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:04.915385 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:04.915462 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:04.915892 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.916462 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.917265 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:04.917499 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:04.917576 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:04.920497 (+ 2921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.129:48383: BlockingRecv error: recv got EOF from 127.1.183.129:48383 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":130,"threads_started":1}
Aug 06 02:46:05 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:05 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:05.123211 15379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:05.116531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.130:42923)
0806 02:46:05.116913 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:05.116932 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:05.117104 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:05.117430 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:05.117433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:05.117817 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:05.117949 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:05.117957 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:05.118834 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:05.118838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:05.119430 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:05.119439 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:05.119945 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:05.119949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:05.120040 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:05.120733 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:05.120746 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:05.123066 (+ 2320us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":126,"threads_started":1}
W20260806 02:46:05.123603 15378 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:05.116705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:42923 (local address 127.1.183.190:35427)
0806 02:46:05.116918 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:05.116926 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:05.117034 (+ 108us) server_negotiation.cc:407] Connection header received
0806 02:46:05.117218 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:05.117221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:05.117277 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:05.117369 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:05.118096 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:05.118677 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:05.119568 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:05.119807 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:05.119922 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:05.123290 (+ 3368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.130:42923: BlockingRecv error: recv got EOF from 127.1.183.130:42923 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Aug 06 02:46:05 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:05 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:05.333693 15380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:05.328099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.131:55537)
0806 02:46:05.328439 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:05.328453 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:05.328557 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:05.328813 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:05.328815 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:05.329019 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:05.329121 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:05.329126 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:05.329875 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:05.329878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:05.330459 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:05.330467 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:05.330950 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:05.330952 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:05.331015 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:05.331624 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:05.331637 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:05.333556 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":157,"threads_started":1}
W20260806 02:46:05.333896 15378 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:05.328249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:55537 (local address 127.1.183.190:35427)
0806 02:46:05.328402 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:05.328406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:05.328548 (+ 142us) server_negotiation.cc:407] Connection header received
0806 02:46:05.328644 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:05.328646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:05.328703 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:05.328770 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:05.329230 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:05.329773 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:05.330590 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:05.330825 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:05.330916 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:05.333734 (+ 2818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.131:55537: BlockingRecv error: recv got EOF from 127.1.183.131:55537 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 06 02:46:05 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:05 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:05.927484 15381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:05.921372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.129:55003)
0806 02:46:05.921846 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:05.921861 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:05.921972 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:05.922280 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:05.922283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:05.922531 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:05.922651 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:05.922657 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:05.923472 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:05.923475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:05.924053 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:05.924063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:05.924606 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:05.924610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:05.924700 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:05.925304 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:05.925316 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:05.927267 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":97,"threads_started":1}
W20260806 02:46:05.927596 15382 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:05.921448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:55003 (local address 127.1.183.190:35427)
0806 02:46:05.921762 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:05.921770 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:05.921942 (+ 172us) server_negotiation.cc:407] Connection header received
0806 02:46:05.922075 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:05.922078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:05.922133 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:05.922247 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:05.922778 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:05.923342 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:05.924214 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:05.924469 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:05.924588 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:05.927442 (+ 2854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.129:55003: BlockingRecv error: recv got EOF from 127.1.183.129:55003 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":106,"threads_started":1}
Aug 06 02:46:06 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:06 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:06.130112 15383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:06.124246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.130:52143)
0806 02:46:06.124565 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:06.124580 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:06.124692 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:06.124906 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:06.124908 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:06.125107 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:06.125206 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:06.125211 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:06.125985 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:06.125990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:06.126758 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:06.126773 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:06.127371 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:06.127375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:06.127467 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:06.127985 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:06.127997 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:06.129957 (+ 1960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":123,"threads_started":1}
W20260806 02:46:06.130245 15382 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:06.124336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:52143 (local address 127.1.183.190:35427)
0806 02:46:06.124507 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:06.124513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:06.124658 (+ 145us) server_negotiation.cc:407] Connection header received
0806 02:46:06.124779 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:06.124782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:06.124830 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:06.124894 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:06.125310 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:06.125847 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:06.126889 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:06.127219 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:06.127328 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:06.130118 (+ 2790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.130:52143: BlockingRecv error: recv got EOF from 127.1.183.130:52143 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 06 02:46:06 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:06 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:06.340685 15384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:06.334645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.131:40701)
0806 02:46:06.335044 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:06.335058 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:06.335195 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:06.335450 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:06.335452 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:06.335653 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:06.335750 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:06.335756 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:06.336594 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:06.336598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:06.337165 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:06.337173 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:06.337619 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:06.337622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:06.337690 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:06.338227 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:06.338239 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:06.340527 (+ 2288us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":114,"threads_started":1}
W20260806 02:46:06.340936 15382 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:06.334871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:40701 (local address 127.1.183.190:35427)
0806 02:46:06.335016 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:06.335021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:06.335145 (+ 124us) server_negotiation.cc:407] Connection header received
0806 02:46:06.335287 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:06.335289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:06.335338 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:06.335405 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:06.335856 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:06.336465 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:06.337285 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:06.337505 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:06.337587 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:06.340718 (+ 3131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.131:40701: BlockingRecv error: recv got EOF from 127.1.183.131:40701 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 06 02:46:06 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:06 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:06.935102 15385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:06.928662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.129:50691)
0806 02:46:06.929008 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:06.929023 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:06.929135 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:06.929426 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:06.929429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:06.929623 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:06.929739 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:06.929745 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:06.930674 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:06.930678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:06.931525 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:06.931539 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:06.932208 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:06.932214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:06.932307 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:06.932949 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:06.932968 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:06.934962 (+ 1994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":98,"threads_started":1}
W20260806 02:46:06.935256 15386 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:06.928839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:50691 (local address 127.1.183.190:35427)
0806 02:46:06.929108 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:06.929113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:06.929128 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:46:06.929216 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:06.929218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:06.929261 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:06.929344 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:06.929876 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:06.930522 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:06.931666 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:06.932001 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:06.932089 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:06.935119 (+ 3030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.129:50691: BlockingRecv error: recv got EOF from 127.1.183.129:50691 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":66,"threads_started":1}
Aug 06 02:46:07 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:07 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:07.136821 15387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:07.131182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.130:43151)
0806 02:46:07.131483 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:07.131497 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:07.131603 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:07.131866 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:07.131868 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:07.132061 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:07.132200 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:07.132206 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.132968 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.132971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:07.133518 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:07.133526 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.133937 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.133940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:07.134004 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:07.134526 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:07.134538 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:07.136654 (+ 2116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":82,"threads_started":1}
W20260806 02:46:07.137090 15386 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:07.131355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:43151 (local address 127.1.183.190:35427)
0806 02:46:07.131558 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:07.131562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:07.131578 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:46:07.131692 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:07.131694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:07.131739 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:07.131802 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:07.132325 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.132867 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.133622 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.133836 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.133913 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:07.136836 (+ 2923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.130:43151: BlockingRecv error: recv got EOF from 127.1.183.130:43151 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 06 02:46:07 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:07 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:07.347980 15388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:07.341868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.131:52097)
0806 02:46:07.342196 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:07.342211 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:07.342367 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:07.342780 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:07.342783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:07.343038 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:07.343167 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:07.343177 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.344018 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.344022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:07.344634 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:07.344643 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.345137 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.345141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:07.345229 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:07.345748 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:07.345760 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:07.347767 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":95,"threads_started":1}
W20260806 02:46:07.348055 15386 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:07.342070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:52097 (local address 127.1.183.190:35427)
0806 02:46:07.342214 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:07.342218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:07.342329 (+ 111us) server_negotiation.cc:407] Connection header received
0806 02:46:07.342551 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:07.342557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:07.342631 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:07.342746 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:07.343278 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.343893 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.344764 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.345011 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.345128 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:07.347924 (+ 2796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.131:52097: BlockingRecv error: recv got EOF from 127.1.183.131:52097 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 06 02:46:07 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:07 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:07.946157 15390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:07.936312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.129:44885)
0806 02:46:07.936632 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:07.936648 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:07.936779 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:07.937079 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:07.937082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:07.937268 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:07.937387 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:07.937392 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.938200 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.938203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:07.938786 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:07.938794 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.939287 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.939289 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:07.939358 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:07.943801 (+ 4443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:07.943825 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:07.946029 (+ 2204us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":113,"threads_started":1}
W20260806 02:46:07.946379 15391 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:07.936373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:44885 (local address 127.1.183.190:35427)
0806 02:46:07.936644 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:07.936649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:07.936739 (+ 90us) server_negotiation.cc:407] Connection header received
0806 02:46:07.936868 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:07.936870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:07.936940 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:07.937022 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:07.937502 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.938069 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.938928 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:07.939168 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:07.939254 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:07.946207 (+ 6953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.129:44885: BlockingRecv error: recv got EOF from 127.1.183.129:44885 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":66,"threads_started":1}
Aug 06 02:46:08 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:08 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:08.143810 15392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:08.137612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.130:54759)
0806 02:46:08.137992 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:08.138006 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:08.138152 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:08.138545 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:08.138552 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:08.138838 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:08.139043 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:08.139054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:08.139918 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:08.139921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:08.140559 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:08.140569 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:08.141018 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:08.141021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:08.141087 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:08.141662 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:08.141675 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:08.143678 (+ 2003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":117,"threads_started":1}
W20260806 02:46:08.144275 15391 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:08.137747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:54759 (local address 127.1.183.190:35427)
0806 02:46:08.137932 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:08.137937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:08.138087 (+ 150us) server_negotiation.cc:407] Connection header received
0806 02:46:08.138296 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:08.138299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:08.138380 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:08.138493 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:08.139191 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:08.139800 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:08.140670 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:08.140913 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:08.140996 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:08.143879 (+ 2883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.130:54759: BlockingRecv error: recv got EOF from 127.1.183.130:54759 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 06 02:46:08 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:08 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:08.355356 15394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:08.349126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.131:46081)
0806 02:46:08.349430 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:08.349443 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:08.349552 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:08.349807 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:08.349809 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:08.350067 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:08.350181 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:08.350188 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:08.351030 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:08.351040 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:08.351681 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:08.351691 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:08.352230 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:08.352233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:08.352297 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:08.352950 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:08.352962 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:08.355171 (+ 2209us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":99,"threads_started":1}
W20260806 02:46:08.355450 15391 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:08.349216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:46081 (local address 127.1.183.190:35427)
0806 02:46:08.349385 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:08.349390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:08.349517 (+ 127us) server_negotiation.cc:407] Connection header received
0806 02:46:08.349636 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:08.349638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:08.349688 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:08.349769 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:08.350294 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:08.350901 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:08.351794 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:08.352060 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:08.352146 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:08.355310 (+ 3164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.131:46081: BlockingRecv error: recv got EOF from 127.1.183.131:46081 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 06 02:46:08 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:08 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:08.953640 15395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:08.947199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.129:39237)
0806 02:46:08.947535 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:08.947549 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:08.947649 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:08.947903 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:08.947906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:08.948119 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:08.948260 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:08.948266 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:08.949088 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:08.949091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:08.949951 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:08.949965 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:08.950652 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:08.950656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:08.950765 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:08.951411 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:08.951428 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:08.953514 (+ 2086us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":133,"threads_started":1}
W20260806 02:46:08.953861 15396 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:08.947407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:39237 (local address 127.1.183.190:35427)
0806 02:46:08.947675 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:08.947679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:08.947693 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:46:08.947737 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:08.947739 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:08.947779 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:08.947861 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:08.948377 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:08.948955 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:08.950143 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:08.950463 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:08.950595 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:08.953703 (+ 3108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.129:39237: BlockingRecv error: recv got EOF from 127.1.183.129:39237 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":98,"threads_started":1}
Aug 06 02:46:09 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:09 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:09.151206 15397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.144860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.130:34097)
0806 02:46:09.145167 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:09.145187 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:09.145320 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:09.145656 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:09.145679 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:09.145981 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:09.146112 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:09.146120 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.146890 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.146893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:09.147475 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:09.147483 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.147922 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.147925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:09.147991 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:09.148554 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:09.148566 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:09.151021 (+ 2455us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":114,"threads_started":1}
W20260806 02:46:09.151384 15396 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:09.145014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:34097 (local address 127.1.183.190:35427)
0806 02:46:09.145160 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:09.145168 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:09.145275 (+ 107us) server_negotiation.cc:407] Connection header received
0806 02:46:09.145434 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:09.145438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:09.145495 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:09.145592 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:09.146231 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.146777 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.147586 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.147810 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.147899 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:09.151212 (+ 3313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.130:34097: BlockingRecv error: recv got EOF from 127.1.183.130:34097 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 06 02:46:09 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:09 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:09.362098 15398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.356530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.131:52321)
0806 02:46:09.356825 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:09.356839 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:09.356949 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:09.357201 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:09.357203 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:09.357395 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:09.357494 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:09.357499 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.358264 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.358266 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:09.358837 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:09.358852 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.359342 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.359345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:09.359407 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:09.359939 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:09.359951 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:09.361968 (+ 2017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":104,"threads_started":1}
W20260806 02:46:09.362313 15396 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:09.356604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:52321 (local address 127.1.183.190:35427)
0806 02:46:09.356767 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:09.356771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:09.356924 (+ 153us) server_negotiation.cc:407] Connection header received
0806 02:46:09.357035 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:09.357037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:09.357087 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:09.357163 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:09.357613 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.358158 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.358979 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.359232 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.359322 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:09.362157 (+ 2835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.131:52321: BlockingRecv error: recv got EOF from 127.1.183.131:52321 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 06 02:46:09 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:09 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:09.960554 15399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:09.954544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.129:60013)
0806 02:46:09.954909 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:09.954930 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:09.955046 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:09.955312 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:09.955314 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:09.955511 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:09.955615 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:09.955619 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.956556 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.956561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:09.957147 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:09.957156 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.957644 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.957647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:09.957745 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:09.958277 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:09.958289 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:09.960424 (+ 2135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":80,"threads_started":1}
W20260806 02:46:09.960858 15400 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:09.954735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:60013 (local address 127.1.183.190:35427)
0806 02:46:09.955033 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:09.955037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:09.955053 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:46:09.955128 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:09.955130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:09.955175 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:09.955257 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:09.955754 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.956427 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.957276 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:09.957519 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:09.957642 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:09.960595 (+ 2953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.129:60013: BlockingRecv error: recv got EOF from 127.1.183.129:60013 (error 108)
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":107,"threads_started":1}
Aug 06 02:46:10 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:10 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 02:46:10.157886 15400 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:10.152140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:40015 (local address 127.1.183.190:35427)
0806 02:46:10.152333 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:10.152337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:10.152457 (+ 120us) server_negotiation.cc:407] Connection header received
0806 02:46:10.152588 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:10.152590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:10.152637 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:10.152713 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:10.153168 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:10.153757 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:10.154537 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:10.154791 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:10.154901 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:10.157778 (+ 2877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.130:40015: BlockingRecv error: recv got EOF from 127.1.183.130:40015 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 02:46:10.157876 15401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:10.152047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.130:40015)
0806 02:46:10.152377 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:10.152404 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:10.152504 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:10.152750 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:10.152753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:10.152956 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:10.153056 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:10.153061 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:10.153862 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:10.153865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:10.154421 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:10.154428 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:10.154911 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:10.154914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:10.154980 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:10.155538 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:10.155550 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:10.157632 (+ 2082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":147,"threads_started":1}
Aug 06 02:46:10 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:10 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:10.373376 15404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:10.363148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.131:39509)
0806 02:46:10.363478 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:10.363494 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:10.363691 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:10.364016 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:10.364018 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:10.364246 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:10.364350 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:10.364355 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:10.365313 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:10.365316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:10.365893 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:10.365902 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:10.366348 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:10.366350 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:10.366416 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:10.371148 (+ 4732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:10.371162 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:10.373237 (+ 2075us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":103,"threads_started":1}
W20260806 02:46:10.373551 15400 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:10.363364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:39509 (local address 127.1.183.190:35427)
0806 02:46:10.363530 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:10.363534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:10.363627 (+ 93us) server_negotiation.cc:407] Connection header received
0806 02:46:10.363846 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:10.363849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:10.363900 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:10.363968 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:10.364518 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:10.365183 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:10.366009 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:10.366237 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:10.366323 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:10.373417 (+ 7094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.131:39509: BlockingRecv error: recv got EOF from 127.1.183.131:39509 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 06 02:46:10 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:10 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:10.968328 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:10.961624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.129:45219)
0806 02:46:10.962012 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:10.962027 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:10.962160 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:10.962443 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:10.962444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:10.962641 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:10.962742 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:10.962746 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:10.963613 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:10.963616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:10.964300 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:10.964311 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:10.964802 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:10.964805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:10.964883 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:10.965670 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:10.965690 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:10.968092 (+ 2402us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":110,"threads_started":1}
W20260806 02:46:10.968441 15406 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:10.961843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:45219 (local address 127.1.183.190:35427)
0806 02:46:10.962158 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:10.962164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:10.962186 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:46:10.962260 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:10.962263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:10.962308 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:10.962401 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:10.962851 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:10.963505 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:10.964420 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:10.964666 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:10.964759 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:10.968284 (+ 3525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.129:45219: BlockingRecv error: recv got EOF from 127.1.183.129:45219 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":100,"threads_started":1}
Aug 06 02:46:11 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:11 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:11.165313 15408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:11.159034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.130:43485)
0806 02:46:11.159492 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:11.159510 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:11.159651 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:11.159964 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:11.159967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:11.160268 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:11.160370 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:11.160376 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:11.161188 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:11.161191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:11.161771 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:11.161779 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:11.162249 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:11.162253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:11.162327 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:11.163026 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:11.163042 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:11.165184 (+ 2142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":118,"threads_started":1}
W20260806 02:46:11.165566 15406 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:11.159109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:43485 (local address 127.1.183.190:35427)
0806 02:46:11.159318 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:11.159323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:11.159622 (+ 299us) server_negotiation.cc:407] Connection header received
0806 02:46:11.159755 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:11.159758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:11.159819 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:11.159917 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:11.160511 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:11.161073 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:11.161877 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:11.162102 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:11.162191 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:11.165439 (+ 3248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.130:43485: BlockingRecv error: recv got EOF from 127.1.183.130:43485 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 06 02:46:11 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:11 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:11.381091 15410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:11.374570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.131:46369)
0806 02:46:11.374951 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:11.374967 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:11.375083 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:11.375371 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:11.375374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:11.375581 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:11.375681 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:11.375687 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:11.376630 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:11.376633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:11.377273 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:11.377282 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:11.377846 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:11.377849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:11.377918 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:11.378506 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:11.378519 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:11.380894 (+ 2375us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":81,"threads_started":1}
W20260806 02:46:11.381157 15406 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:11.374805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:46369 (local address 127.1.183.190:35427)
0806 02:46:11.374991 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:11.374995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:11.375033 (+ 38us) server_negotiation.cc:407] Connection header received
0806 02:46:11.375187 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:11.375190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:11.375242 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:11.375322 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:11.375847 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:11.376520 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:11.377431 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:11.377673 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:11.377824 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:11.381036 (+ 3212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.131:46369: BlockingRecv error: recv got EOF from 127.1.183.131:46369 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 06 02:46:11 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:11 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:11.975057 15411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:11.969349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.129:36515)
0806 02:46:11.969682 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:11.969696 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:11.969793 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:11.970043 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:11.970046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:11.970235 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:11.970339 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:11.970344 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:11.971211 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:11.971213 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:11.971774 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:11.971782 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:11.972327 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:11.972329 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:11.972396 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:11.972947 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:11.972960 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:11.974910 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":105,"threads_started":1}
W20260806 02:46:11.975234 15412 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:11.969505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:36515 (local address 127.1.183.190:35427)
0806 02:46:11.969811 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:11.969815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:11.969830 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:46:11.969870 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:11.969873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:11.969917 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:11.970003 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:11.970466 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:11.971095 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:11.971892 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:11.972138 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:11.972323 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:11.975087 (+ 2764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.129:36515: BlockingRecv error: recv got EOF from 127.1.183.129:36515 (error 108)
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":74,"threads_started":1}
Aug 06 02:46:12 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:12 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:12.172294 15414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:12.166169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.130:54317)
0806 02:46:12.166457 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:12.166471 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:12.166576 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:12.166836 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:12.166838 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:12.167041 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:12.167146 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:12.167152 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:12.168233 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:12.168235 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:12.168799 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:12.168807 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:12.169290 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:12.169292 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:12.169366 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:12.169983 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:12.169997 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:12.172093 (+ 2096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":95,"threads_started":1}
W20260806 02:46:12.172449 15412 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:12.166256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:54317 (local address 127.1.183.190:35427)
0806 02:46:12.166442 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:12.166446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:12.166547 (+ 101us) server_negotiation.cc:407] Connection header received
0806 02:46:12.166662 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:12.166665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:12.166715 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:12.166831 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:12.167269 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:12.168066 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:12.168929 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:12.169179 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:12.169272 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:12.172306 (+ 3034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.130:54317: BlockingRecv error: recv got EOF from 127.1.183.130:54317 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 06 02:46:12 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:12 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:12.388695 15417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:12.382135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.131:60499)
0806 02:46:12.382456 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:12.382478 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:12.382622 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:12.383000 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:12.383004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:12.383249 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:12.383423 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:12.383433 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:12.384491 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:12.384494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:12.385091 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:12.385100 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:12.385673 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:12.385677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:12.385749 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:12.386374 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:12.386386 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:12.388565 (+ 2179us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":104,"threads_started":1}
W20260806 02:46:12.388875 15412 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:12.382254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:60499 (local address 127.1.183.190:35427)
0806 02:46:12.382460 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:12.382464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:12.382574 (+ 110us) server_negotiation.cc:407] Connection header received
0806 02:46:12.382747 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:12.382753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:12.382818 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:12.382930 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:12.383559 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:12.384356 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:12.385259 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:12.385538 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:12.385651 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:12.388740 (+ 3089us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.131:60499: BlockingRecv error: recv got EOF from 127.1.183.131:60499 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 06 02:46:12 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:12 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:12.985069 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:12.976023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.129:56021)
0806 02:46:12.976398 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:12.976417 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:12.976537 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:12.976794 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:12.976796 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:12.976997 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:12.977104 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:12.977109 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:12.978093 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:12.978096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:12.978711 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:12.978722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:12.979199 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:12.979202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:12.979270 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:12.982678 (+ 3408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:12.982693 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:12.984907 (+ 2214us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":147,"threads_started":1}
W20260806 02:46:12.985251 15420 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:12.976214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:56021 (local address 127.1.183.190:35427)
0806 02:46:12.976519 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:12.976523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:12.976540 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:46:12.976614 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:12.976616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:12.976659 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:12.976743 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:12.977263 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:12.977937 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:12.978829 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:12.979081 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:12.979165 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:12.985101 (+ 5936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.129:56021: BlockingRecv error: recv got EOF from 127.1.183.129:56021 (error 108)
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":114,"threads_started":1}
Aug 06 02:46:13 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:13 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:13.180207 15422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:13.173335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.130:52821)
0806 02:46:13.173664 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:13.173678 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:13.173786 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:13.174057 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:13.174059 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:13.174251 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:13.174366 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:13.174372 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:13.175311 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:13.175357 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:13.176373 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:13.176389 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:13.176997 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:13.177000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:13.177071 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:13.177603 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:13.177616 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:13.179969 (+ 2353us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":111,"threads_started":1}
W20260806 02:46:13.180243 15420 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:13.173514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:52821 (local address 127.1.183.190:35427)
0806 02:46:13.173723 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:13.173727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:13.173745 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:46:13.173887 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:13.173889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:13.173937 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:13.174013 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:13.174473 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:13.175150 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:13.176507 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:13.176877 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:13.176968 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:13.180124 (+ 3156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.130:52821: BlockingRecv error: recv got EOF from 127.1.183.130:52821 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Aug 06 02:46:13 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:13 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:13.395895 15423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:13.389721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.131:50387)
0806 02:46:13.390054 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:13.390068 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:13.390170 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:13.390516 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:13.390519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:13.390714 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:13.390847 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:13.390853 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:13.391649 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:13.391651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:13.392353 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:13.392364 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:13.392876 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:13.392879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:13.392946 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:13.393546 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:13.393558 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:13.395650 (+ 2092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":121,"threads_started":1}
W20260806 02:46:13.395972 15420 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:13.389930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:50387 (local address 127.1.183.190:35427)
0806 02:46:13.390091 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:13.390095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:13.390123 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:46:13.390317 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:13.390320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:13.390373 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:13.390451 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:13.390967 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:13.391495 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:13.392504 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:13.392732 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:13.392854 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:13.395825 (+ 2971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.131:50387: BlockingRecv error: recv got EOF from 127.1.183.131:50387 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 06 02:46:13 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:13 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:13.991588 15424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:13.986040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.129:47291)
0806 02:46:13.986332 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:13.986346 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:13.986444 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:13.986718 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:13.986720 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:13.986909 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:13.987014 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:13.987019 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:13.987770 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:13.987772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:13.988384 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:13.988394 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:13.988835 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:13.988838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:13.988901 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:13.989501 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:13.989513 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:13.991459 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":93,"threads_started":1}
W20260806 02:46:13.991757 15425 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:13.986082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:47291 (local address 127.1.183.190:35427)
0806 02:46:13.986338 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:13.986342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:13.986419 (+ 77us) server_negotiation.cc:407] Connection header received
0806 02:46:13.986524 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:13.986542 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:13.986587 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:13.986670 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:13.987116 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:13.987669 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:13.988492 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:13.988727 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:13.988813 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:13.991613 (+ 2800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.129:47291: BlockingRecv error: recv got EOF from 127.1.183.129:47291 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":100,"threads_started":1}
Aug 06 02:46:14 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:14 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:14.187120 15426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:14.181261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.130:34567)
0806 02:46:14.181592 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:14.181606 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:14.181744 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:14.182047 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:14.182051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:14.182282 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:14.182404 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:14.182409 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:14.183199 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:14.183204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:14.183781 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:14.183796 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:14.184324 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:14.184327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:14.184416 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:14.184988 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:14.185000 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:14.186932 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":97,"threads_started":1}
W20260806 02:46:14.187199 15425 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:14.181331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:34567 (local address 127.1.183.190:35427)
0806 02:46:14.181521 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:14.181526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:14.181686 (+ 160us) server_negotiation.cc:407] Connection header received
0806 02:46:14.181866 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:14.181870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:14.181927 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:14.182022 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:14.182527 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:14.183079 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:14.183911 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:14.184147 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:14.184324 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:14.187085 (+ 2761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.130:34567: BlockingRecv error: recv got EOF from 127.1.183.130:34567 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 06 02:46:14 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:14 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:14.402913 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:14.396856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.131:60219)
0806 02:46:14.397170 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:14.397185 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:14.397301 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:14.397561 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:14.397563 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:14.397776 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:14.397883 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:14.397889 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:14.398676 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:14.398679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:14.399322 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:14.399332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:14.399782 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:14.399785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:14.399856 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:14.400457 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:14.400471 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:14.402782 (+ 2311us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":119,"threads_started":1}
W20260806 02:46:14.403092 15425 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:14.396966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:60219 (local address 127.1.183.190:35427)
0806 02:46:14.397153 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:14.397157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:14.397257 (+ 100us) server_negotiation.cc:407] Connection header received
0806 02:46:14.397383 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:14.397385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:14.397443 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:14.397517 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:14.397994 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:14.398551 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:14.399443 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:14.399664 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:14.399741 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:14.402946 (+ 3205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.131:60219: BlockingRecv error: recv got EOF from 127.1.183.131:60219 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 06 02:46:14 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:14 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:14.998607 15429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:14.992821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.129:39489)
0806 02:46:14.993140 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:14.993155 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:14.993253 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:14.993508 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:14.993510 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:14.993697 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:14.993798 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:14.993802 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:14.994550 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:14.994553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:14.995232 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:14.995245 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:14.995757 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:14.995760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:14.995845 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:14.996466 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:14.996478 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:14.998413 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":126,"threads_started":1}
W20260806 02:46:14.998703 15430 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:14.992931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:39489 (local address 127.1.183.190:35427)
0806 02:46:14.993188 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:14.993192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:14.993206 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:46:14.993328 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:14.993330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:14.993373 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:14.993456 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:14.993907 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:14.994451 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:14.995381 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:14.995631 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:14.995750 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:14.998566 (+ 2816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.129:39489: BlockingRecv error: recv got EOF from 127.1.183.129:39489 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":91,"threads_started":1}
Aug 06 02:46:15 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:15 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:15.194505 15431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:15.188434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.130:49131)
0806 02:46:15.188794 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:15.188808 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:15.188957 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:15.189246 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:15.189250 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:15.189499 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:15.189622 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:15.189631 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.190673 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.190678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:15.191411 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:15.191421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.191870 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.191873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:15.191938 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:15.192475 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:15.192487 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:15.194381 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":81,"threads_started":1}
W20260806 02:46:15.194731 15430 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:15.188500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:49131 (local address 127.1.183.190:35427)
0806 02:46:15.188712 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:15.188718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:15.188899 (+ 181us) server_negotiation.cc:407] Connection header received
0806 02:46:15.189078 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:15.189081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:15.189137 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:15.189223 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:15.189744 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.190554 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.191532 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.191764 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.191853 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:15.194595 (+ 2742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.130:49131: BlockingRecv error: recv got EOF from 127.1.183.130:49131 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 06 02:46:15 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:15 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:15.410075 15432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:15.403998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.131:36581)
0806 02:46:15.404333 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:15.404347 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:15.404455 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:15.404708 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:15.404710 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:15.404910 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:15.405012 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:15.405018 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.405864 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.405868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:15.406450 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:15.406459 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.406983 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.406986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:15.407055 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:15.407660 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:15.407677 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:15.409886 (+ 2209us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":104,"threads_started":1}
W20260806 02:46:15.410135 15430 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:15.404081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:36581 (local address 127.1.183.190:35427)
0806 02:46:15.404285 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:15.404289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:15.404421 (+ 132us) server_negotiation.cc:407] Connection header received
0806 02:46:15.404535 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:15.404538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:15.404586 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:15.404666 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:15.405156 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.405709 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.406583 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:15.406861 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:15.406958 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:15.410020 (+ 3062us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.131:36581: BlockingRecv error: recv got EOF from 127.1.183.131:36581 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Aug 06 02:46:16 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:16 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:16.005622 15435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:15.999707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.129:59255)
0806 02:46:16.000015 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:16.000042 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:16.000144 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:16.000443 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:16.000445 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:16.000635 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:16.000736 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:16.000741 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:16.001519 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:16.001521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:16.002079 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:16.002087 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:16.002569 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:16.002572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:16.002635 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:16.003166 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:16.003182 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:16.005500 (+ 2318us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":96,"threads_started":1}
W20260806 02:46:16.005944 15437 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:15.999843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:59255 (local address 127.1.183.190:35427)
0806 02:46:16.000091 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:16.000095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:16.000111 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:46:16.000262 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:16.000265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:16.000317 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:16.000398 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:16.000861 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:16.001414 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:16.002234 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:16.002466 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:16.002545 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:16.005655 (+ 3110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.129:59255: BlockingRecv error: recv got EOF from 127.1.183.129:59255 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":68,"threads_started":1}
Aug 06 02:46:16 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:16 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:16.201589 15438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:16.195362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.130:41505)
0806 02:46:16.195740 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:16.195762 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:16.195923 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:16.196244 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:16.196246 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:16.196466 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:16.196565 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:16.196571 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:16.197343 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:16.197349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:16.198011 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:16.198024 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:16.198672 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:16.198675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:16.198778 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:16.199359 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:16.199374 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:16.201467 (+ 2093us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":101,"threads_started":1}
W20260806 02:46:16.201778 15437 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:16.195539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:41505 (local address 127.1.183.190:35427)
0806 02:46:16.195714 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:16.195720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:16.195837 (+ 117us) server_negotiation.cc:407] Connection header received
0806 02:46:16.196039 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:16.196043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:16.196099 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:16.196221 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:16.196688 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:16.197228 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:16.198143 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:16.198506 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:16.198635 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:16.201611 (+ 2976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.130:41505: BlockingRecv error: recv got EOF from 127.1.183.130:41505 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 06 02:46:16 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:16 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:16.418615 15439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:16.411067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.131:56081)
0806 02:46:16.411412 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:16.411426 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:16.411573 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:16.411858 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:16.411861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:16.412113 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:16.412296 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:16.412304 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:16.413455 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:16.413459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:16.414539 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:16.414551 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:16.415258 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:16.415262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:16.415350 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:16.415984 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:16.416002 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:16.418407 (+ 2405us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":113,"threads_started":1}
W20260806 02:46:16.418648 15437 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:16.411162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:56081 (local address 127.1.183.190:35427)
0806 02:46:16.411352 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:16.411357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:16.411511 (+ 154us) server_negotiation.cc:407] Connection header received
0806 02:46:16.411680 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:16.411683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:16.411739 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:16.411832 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:16.412433 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:16.413311 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:16.414672 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:16.415122 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:16.415245 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:16.418542 (+ 3297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.131:56081: BlockingRecv error: recv got EOF from 127.1.183.131:56081 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 06 02:46:17 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:17 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:17.012405 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:17.006484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.129:33571)
0806 02:46:17.006790 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:17.006805 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:17.006901 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:17.007169 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:17.007171 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:17.007387 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:17.007491 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:17.007496 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:17.008306 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:17.008310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:17.008932 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:17.008941 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:17.009498 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:17.009502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:17.009603 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:17.010154 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:17.010166 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:17.012209 (+ 2043us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":115,"threads_started":1}
W20260806 02:46:17.012738 15443 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:17.006578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:33571 (local address 127.1.183.190:35427)
0806 02:46:17.006827 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:17.006831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:17.006848 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:46:17.006991 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:17.006993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:17.007039 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:17.007121 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:17.007609 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:17.008145 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:17.009067 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:17.009317 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:17.009471 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:17.012341 (+ 2870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.129:33571: BlockingRecv error: recv got EOF from 127.1.183.129:33571 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":89,"threads_started":1}
Aug 06 02:46:17 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:17 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:17.208573 15445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:17.202452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.130:35709)
0806 02:46:17.202819 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:17.202837 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:17.203009 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:17.203309 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:17.203311 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:17.203543 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:17.203645 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:17.203651 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:17.204538 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:17.204543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:17.205129 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:17.205137 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:17.205624 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:17.205628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:17.205718 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:17.206319 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:17.206330 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:17.208396 (+ 2066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":132,"threads_started":1}
W20260806 02:46:17.208680 15443 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:17.202648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:35709 (local address 127.1.183.190:35427)
0806 02:46:17.202820 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:17.202827 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:17.202923 (+ 96us) server_negotiation.cc:407] Connection header received
0806 02:46:17.203128 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:17.203131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:17.203188 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:17.203279 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:17.203776 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:17.204412 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:17.205257 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:17.205504 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:17.205618 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:17.208560 (+ 2942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.130:35709: BlockingRecv error: recv got EOF from 127.1.183.130:35709 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 06 02:46:17 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:17 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:17.425968 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:17.419851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.131:34645)
0806 02:46:17.420182 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:17.420196 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:17.420341 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:17.420666 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:17.420669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:17.420899 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:17.420998 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:17.421004 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:17.421794 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:17.421799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:17.422398 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:17.422407 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:17.422881 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:17.422885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:17.422977 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:17.423624 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:17.423636 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:17.425741 (+ 2105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":94,"threads_started":1}
W20260806 02:46:17.425983 15443 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:17.419918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:34645 (local address 127.1.183.190:35427)
0806 02:46:17.420080 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:17.420085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:17.420288 (+ 203us) server_negotiation.cc:407] Connection header received
0806 02:46:17.420475 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:17.420479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:17.420539 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:17.420637 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:17.421137 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:17.421678 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:17.422527 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:17.422761 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:17.422873 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:17.425876 (+ 3003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.131:34645: BlockingRecv error: recv got EOF from 127.1.183.131:34645 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 06 02:46:18 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:18 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:18.022745 15447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:18.013353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.129:57515)
0806 02:46:18.013695 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:18.013713 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:18.013836 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:18.014083 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:18.014085 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:18.014271 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:18.014376 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:18.014381 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:18.015142 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:18.015145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:18.015729 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:18.015737 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:18.016255 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:18.016258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:18.016325 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:18.020508 (+ 4183us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:18.020522 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:18.022605 (+ 2083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":102,"threads_started":1}
W20260806 02:46:18.023001 15448 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:18.013476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:57515 (local address 127.1.183.190:35427)
0806 02:46:18.013858 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:18.013862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:18.013876 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:46:18.013915 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:18.013917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:18.013958 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:18.014038 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:18.014488 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:18.015027 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:18.015849 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:18.016087 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:18.016232 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:18.022832 (+ 6600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.129:57515: BlockingRecv error: recv got EOF from 127.1.183.129:57515 (error 108)
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":85,"threads_started":1}
Aug 06 02:46:18 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:18 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:18.216337 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:18.209687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.130:44639)
0806 02:46:18.210021 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:18.210036 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:18.210191 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:18.210491 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:18.210493 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:18.210714 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:18.210813 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:18.210820 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:18.211609 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:18.211614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:18.212235 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:18.212244 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:18.212727 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:18.212730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:18.212818 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:18.213521 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:18.213537 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:18.216040 (+ 2503us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":117,"threads_started":1}
W20260806 02:46:18.216444 15448 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:18.209800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:44639 (local address 127.1.183.190:35427)
0806 02:46:18.209974 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:18.209979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:18.210132 (+ 153us) server_negotiation.cc:407] Connection header received
0806 02:46:18.210306 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:18.210317 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:18.210371 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:18.210466 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:18.210945 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:18.211489 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:18.212367 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:18.212601 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:18.212717 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:18.216302 (+ 3585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.130:44639: BlockingRecv error: recv got EOF from 127.1.183.130:44639 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 06 02:46:18 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:18 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.131@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:18.433262 15451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:18.427079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.131:49007)
0806 02:46:18.427511 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:18.427524 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:18.427636 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:18.427908 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:18.427910 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:18.428102 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:18.428280 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:18.428286 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:18.429238 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:18.429243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:18.429845 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:18.429854 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:18.430383 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:18.430386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:18.430479 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:18.431044 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:18.431057 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:18.433125 (+ 2068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":123,"threads_started":1}
W20260806 02:46:18.433467 15448 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:18.427249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:49007 (local address 127.1.183.190:35427)
0806 02:46:18.427420 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:18.427424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:18.427605 (+ 181us) server_negotiation.cc:407] Connection header received
0806 02:46:18.427733 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:18.427735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:18.427786 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:18.427852 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:18.428457 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:18.429093 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:18.430009 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:18.430251 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:18.430374 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:18.433285 (+ 2911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.131:49007: BlockingRecv error: recv got EOF from 127.1.183.131:49007 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 06 02:46:19 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:19 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.129@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:19.029634 15452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:19.023727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.129:38601)
0806 02:46:19.024047 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:19.024061 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:19.024173 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:19.024440 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:19.024442 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:19.024633 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:19.024744 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:19.024750 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:19.025558 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:19.025561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:19.026198 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:19.026208 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:19.026744 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:19.026747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:19.026827 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:19.027388 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:19.027401 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:19.029493 (+ 2092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":102,"threads_started":1}
W20260806 02:46:19.029881 15453 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:19.023825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:38601 (local address 127.1.183.190:35427)
0806 02:46:19.024113 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:19.024117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:19.024131 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:46:19.024263 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:19.024265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:19.024321 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:19.024400 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:19.024888 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:19.025445 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:19.026345 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:19.026594 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:19.026680 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:19.029707 (+ 3027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.129:38601: BlockingRecv error: recv got EOF from 127.1.183.129:38601 (error 108)
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":100,"threads_started":1}
Aug 06 02:46:19 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:46:19 dist-test-slave-gmjp krb5kdc[13906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.183.130@KRBTEST.COM for oryx/127.1.183.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:46:19.223348 15454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:19.217404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35427 (local address 127.1.183.130:59033)
0806 02:46:19.217896 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:19.217913 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:19.218026 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:19.218275 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:19.218277 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:19.218483 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:19.218582 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:19.218587 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:19.219449 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:19.219452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:19.220023 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:19.220031 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:19.220742 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:19.220745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:19.220814 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:19.221310 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:19.221323 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:19.223201 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.183.190:35427: Server oryx/127.1.183.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":86,"threads_started":1}
W20260806 02:46:19.223533 15453 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:19.217504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:59033 (local address 127.1.183.190:35427)
0806 02:46:19.217683 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:19.217688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:19.217987 (+ 299us) server_negotiation.cc:407] Connection header received
0806 02:46:19.218096 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:19.218098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:19.218174 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:19.218262 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:19.218681 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:19.219339 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:19.220137 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:19.220634 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:19.220720 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:19.223351 (+ 2631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.183.130:59033: BlockingRecv error: recv got EOF from 127.1.183.130:59033 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:46:19.311846 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 14757
I20260806 02:46:19.325992 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 14992
I20260806 02:46:19.339435 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 15179
I20260806 02:46:19.355444 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 14639
2026-08-06T02:46:19Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (19382 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-06T02:46:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:46:19Z Disabled control of system clock
I20260806 02:46:19.395406 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:40253
--webserver_interface=127.1.183.190
--webserver_port=0
--builtin_ntp_servers=127.1.183.148:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:40253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 02:46:19.519464 15460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:19.519783 15460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:19.519821 15460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:19.523864 15460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:46:19.523934 15460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:19.523950 15460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:19.523967 15460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:46:19.523983 15460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:46:19.528769 15460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:40253
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:40253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:46:19.529881 15460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:19.531280 15460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:19.540688 15466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:19.540714 15467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:19.540688 15469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:19.541026 15460 server_base.cc:1034] running on GCE node
I20260806 02:46:19.541862 15460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:19.543058 15460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:19.544245 15460 hybrid_clock.cc:648] HybridClock initialized: now 1785984379544213 us; error 45 us; skew 500 ppm
I20260806 02:46:19.546311 15460 webserver.cc:460] Webserver started at http://127.1.183.190:46319/ using document root <none> and password file <none>
I20260806 02:46:19.546963 15460 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:19.547068 15460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:19.547319 15460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:19.549610 15460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "e6e911842ac144dbb00d9debfa9c9793"
format_stamp: "Formatted at 2026-08-06 02:46:19 on dist-test-slave-gmjp"
server_key: "9feed31601bd162db90b5350baa80ae8"
server_key_iv: "8894746bdeb563541539ea67cdf36754"
server_key_version: "encryptionkey@0"
I20260806 02:46:19.550235 15460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "e6e911842ac144dbb00d9debfa9c9793"
format_stamp: "Formatted at 2026-08-06 02:46:19 on dist-test-slave-gmjp"
server_key: "9feed31601bd162db90b5350baa80ae8"
server_key_iv: "8894746bdeb563541539ea67cdf36754"
server_key_version: "encryptionkey@0"
I20260806 02:46:19.554368 15460 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:46:19.556980 15479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:19.558255 15460 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:46:19.558411 15460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "e6e911842ac144dbb00d9debfa9c9793"
format_stamp: "Formatted at 2026-08-06 02:46:19 on dist-test-slave-gmjp"
server_key: "9feed31601bd162db90b5350baa80ae8"
server_key_iv: "8894746bdeb563541539ea67cdf36754"
server_key_version: "encryptionkey@0"
I20260806 02:46:19.558548 15460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:19.580583 15460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:19.581413 15460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:19.581640 15460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:19.590312 15460 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:40253
I20260806 02:46:19.590319 15560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:40253 every 8 connection(s)
I20260806 02:46:19.591406 15460 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:46:19.592768 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 15460
I20260806 02:46:19.592908 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:46:19.593201 1758 external_mini_cluster.cc:1428] Setting key b5c4f93c2b973c079321797a908220c2
I20260806 02:46:19.595564 15561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:19.601555 15561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6e911842ac144dbb00d9debfa9c9793: Bootstrap starting.
I20260806 02:46:19.603264 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:19.594733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46938 (local address 127.1.183.190:40253)
0806 02:46:19.595441 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:19.595458 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:19.595496 (+ 38us) server_negotiation.cc:407] Connection header received
0806 02:46:19.596303 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:19.596322 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:19.596740 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:19.597103 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:19.597552 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:19.598589 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:19.599646 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:19.600206 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:19.600423 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:19.601376 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:19.601381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:19.601395 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:46:19.601864 (+ 469us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:19.601927 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:46:19.601949 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:19.602058 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:19.602232 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 02:46:19.602523 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":169,"threads_started":1}
I20260806 02:46:19.604264 15561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6e911842ac144dbb00d9debfa9c9793: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:19.605201 15561 log.cc:824] T 00000000000000000000000000000000 P e6e911842ac144dbb00d9debfa9c9793: Log is configured to *not* fsync() on all Append() calls
I20260806 02:46:19.607483 15561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6e911842ac144dbb00d9debfa9c9793: No bootstrap required, opened a new log
I20260806 02:46:19.611258 15561 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6e911842ac144dbb00d9debfa9c9793 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e911842ac144dbb00d9debfa9c9793" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 40253 } }
I20260806 02:46:19.611603 15561 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6e911842ac144dbb00d9debfa9c9793 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:19.611686 15561 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6e911842ac144dbb00d9debfa9c9793 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6e911842ac144dbb00d9debfa9c9793, State: Initialized, Role: FOLLOWER
I20260806 02:46:19.612218 15561 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6e911842ac144dbb00d9debfa9c9793 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e911842ac144dbb00d9debfa9c9793" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 40253 } }
I20260806 02:46:19.612392 15561 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6e911842ac144dbb00d9debfa9c9793 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:46:19.612493 15561 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6e911842ac144dbb00d9debfa9c9793 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:46:19.612625 15561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6e911842ac144dbb00d9debfa9c9793 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:19.613658 15561 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6e911842ac144dbb00d9debfa9c9793 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e911842ac144dbb00d9debfa9c9793" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 40253 } }
I20260806 02:46:19.614041 15561 leader_election.cc:302] T 00000000000000000000000000000000 P e6e911842ac144dbb00d9debfa9c9793 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6e911842ac144dbb00d9debfa9c9793; no voters:
I20260806 02:46:19.614385 15561 leader_election.cc:288] T 00000000000000000000000000000000 P e6e911842ac144dbb00d9debfa9c9793 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:46:19.614645 15568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6e911842ac144dbb00d9debfa9c9793 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:46:19.614944 15568 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6e911842ac144dbb00d9debfa9c9793 [term 1 LEADER]: Becoming Leader. State: Replica: e6e911842ac144dbb00d9debfa9c9793, State: Running, Role: LEADER
I20260806 02:46:19.615259 15568 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6e911842ac144dbb00d9debfa9c9793 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e911842ac144dbb00d9debfa9c9793" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 40253 } }
I20260806 02:46:19.615787 15561 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6e911842ac144dbb00d9debfa9c9793 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:46:19.616918 15571 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6e911842ac144dbb00d9debfa9c9793 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6e911842ac144dbb00d9debfa9c9793" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e911842ac144dbb00d9debfa9c9793" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 40253 } } }
I20260806 02:46:19.617059 15571 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6e911842ac144dbb00d9debfa9c9793 [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:19.617308 15572 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6e911842ac144dbb00d9debfa9c9793 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6e911842ac144dbb00d9debfa9c9793. Latest consensus state: current_term: 1 leader_uuid: "e6e911842ac144dbb00d9debfa9c9793" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e911842ac144dbb00d9debfa9c9793" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 40253 } } }
I20260806 02:46:19.617458 15572 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6e911842ac144dbb00d9debfa9c9793 [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:19.617483 15581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:46:19.621304 15581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:46:19.627288 15581 catalog_manager.cc:1347] Generated new cluster ID: 67099296876446448525dfb103d05dc3
I20260806 02:46:19.627405 15581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:46:19.632524 15581 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:46:19.633500 15581 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:46:19.639083 15581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6e911842ac144dbb00d9debfa9c9793: Generated new TSK 0
I20260806 02:46:19.639892 15581 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:46:19.645802 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.129:0
--local_ip_for_outbound_sockets=127.1.183.129
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:40253
--builtin_ntp_servers=127.1.183.148:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260806 02:46:19.769212 15603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:19.769556 15603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:19.769635 15603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:19.773823 15603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:19.773931 15603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:19.774032 15603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.129
I20260806 02:46:19.779139 15603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:40253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.129
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:46:19.780470 15603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:19.782016 15603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:19.792629 15614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:19.792606 15611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:19.792611 15617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:19.792747 15603 server_base.cc:1034] running on GCE node
I20260806 02:46:19.793387 15603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:19.794112 15603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:19.795313 15603 hybrid_clock.cc:648] HybridClock initialized: now 1785984379795291 us; error 35 us; skew 500 ppm
I20260806 02:46:19.797461 15603 webserver.cc:460] Webserver started at http://127.1.183.129:38081/ using document root <none> and password file <none>
I20260806 02:46:19.798120 15603 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:19.798224 15603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:19.798487 15603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:19.800750 15603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/data/instance:
uuid: "b401f0a2619245c9b71dd2cec8ebc470"
format_stamp: "Formatted at 2026-08-06 02:46:19 on dist-test-slave-gmjp"
server_key: "0f53684b9c2cca5a505dcf04c2d634ab"
server_key_iv: "af7c5b4f2abba05ddaa4518ebe104f91"
server_key_version: "encryptionkey@0"
I20260806 02:46:19.801357 15603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance:
uuid: "b401f0a2619245c9b71dd2cec8ebc470"
format_stamp: "Formatted at 2026-08-06 02:46:19 on dist-test-slave-gmjp"
server_key: "0f53684b9c2cca5a505dcf04c2d634ab"
server_key_iv: "af7c5b4f2abba05ddaa4518ebe104f91"
server_key_version: "encryptionkey@0"
I20260806 02:46:19.805414 15603 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260806 02:46:19.808355 15625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:19.809752 15603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:46:19.809991 15603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/wal
uuid: "b401f0a2619245c9b71dd2cec8ebc470"
format_stamp: "Formatted at 2026-08-06 02:46:19 on dist-test-slave-gmjp"
server_key: "0f53684b9c2cca5a505dcf04c2d634ab"
server_key_iv: "af7c5b4f2abba05ddaa4518ebe104f91"
server_key_version: "encryptionkey@0"
I20260806 02:46:19.810133 15603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:19.837167 15603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:19.837963 15603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:19.838181 15603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:19.838832 15603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:19.839836 15603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:19.839906 15603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:19.839980 15603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:19.840021 15603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:19.849862 15603 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.129:44367
I20260806 02:46:19.849888 15782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.129:44367 every 8 connection(s)
I20260806 02:46:19.850870 15603 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
I20260806 02:46:19.854475 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 15603
I20260806 02:46:19.854610 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance
I20260806 02:46:19.854929 1758 external_mini_cluster.cc:1428] Setting key 25794261b606e0707a77e52ee8fc1e81
I20260806 02:46:19.857244 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.130:0
--local_ip_for_outbound_sockets=127.1.183.130
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:40253
--builtin_ntp_servers=127.1.183.148:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 02:46:19.861873 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:19.852975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:58371 (local address 127.1.183.190:40253)
0806 02:46:19.853129 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:19.853132 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:19.854326 (+ 1194us) server_negotiation.cc:407] Connection header received
0806 02:46:19.856214 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:19.856217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:19.856277 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:19.856400 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:19.857659 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:19.858201 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:19.859774 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:19.860016 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:19.860134 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:19.861334 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:19.861338 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:19.861341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:46:19.861362 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:19.861380 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:46:19.861394 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:19.861480 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:19.861645 (+ 165us) server_negotiation.cc:299] Negotiation successful
0806 02:46:19.861724 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:46:19.862859 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:19.853370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:40253 (local address 127.1.183.129:58371)
0806 02:46:19.854088 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:19.854183 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:19.855958 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:19.856616 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:19.856625 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:19.856739 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:19.857485 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:19.857508 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:19.858341 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:19.858348 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:19.859642 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:19.859653 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:19.860225 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:19.860233 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:19.860397 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:19.861055 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:46:19.861057 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:19.861100 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:46:19.861170 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:19.861482 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:19.861495 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:19.861502 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 02:46:19.861646 (+ 144us) client_negotiation.cc:240] Negotiation successful
0806 02:46:19.862049 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":135,"threads_started":1}
I20260806 02:46:19.864243 15783 heartbeater.cc:347] Connected to a master server at 127.1.183.190:40253
I20260806 02:46:19.864539 15783 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:19.865127 15783 heartbeater.cc:511] Master 127.1.183.190:40253 requested a full tablet report, sending...
I20260806 02:46:19.866763 15505 ts_manager.cc:194] Registered new tserver with Master: b401f0a2619245c9b71dd2cec8ebc470 (127.1.183.129:44367)
I20260806 02:46:19.867955 15505 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.183.129:58371
W20260806 02:46:19.977514 15786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:19.977864 15786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:19.977952 15786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:19.982208 15786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:19.982303 15786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:19.982398 15786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.130
I20260806 02:46:19.987512 15786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:40253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.130
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:46:19.988894 15786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:19.990455 15786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:20.000950 15800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:20.000943 15795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:20.000965 15794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:20.002105 15786 server_base.cc:1034] running on GCE node
I20260806 02:46:20.002568 15786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:20.003254 15786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:20.004446 15786 hybrid_clock.cc:648] HybridClock initialized: now 1785984380004433 us; error 33 us; skew 500 ppm
I20260806 02:46:20.006343 15786 webserver.cc:460] Webserver started at http://127.1.183.130:44737/ using document root <none> and password file <none>
I20260806 02:46:20.007000 15786 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:20.007097 15786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:20.007344 15786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:20.009538 15786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/data/instance:
uuid: "5a8518c14ea54ba0b7b615c8b404b2f3"
format_stamp: "Formatted at 2026-08-06 02:46:20 on dist-test-slave-gmjp"
server_key: "0835210857ad2f71c66455dcb3117070"
server_key_iv: "2dd35096410704a86782570774008f66"
server_key_version: "encryptionkey@0"
I20260806 02:46:20.010133 15786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance:
uuid: "5a8518c14ea54ba0b7b615c8b404b2f3"
format_stamp: "Formatted at 2026-08-06 02:46:20 on dist-test-slave-gmjp"
server_key: "0835210857ad2f71c66455dcb3117070"
server_key_iv: "2dd35096410704a86782570774008f66"
server_key_version: "encryptionkey@0"
I20260806 02:46:20.014142 15786 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:46:20.017109 15812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:20.018558 15786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:46:20.018667 15786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/wal
uuid: "5a8518c14ea54ba0b7b615c8b404b2f3"
format_stamp: "Formatted at 2026-08-06 02:46:20 on dist-test-slave-gmjp"
server_key: "0835210857ad2f71c66455dcb3117070"
server_key_iv: "2dd35096410704a86782570774008f66"
server_key_version: "encryptionkey@0"
I20260806 02:46:20.018778 15786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:20.033360 15786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:20.034096 15786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:20.034263 15786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:20.034861 15786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:20.035920 15786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:20.035971 15786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:20.036046 15786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:20.036085 15786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:20.045567 15786 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.130:46659
I20260806 02:46:20.045848 15977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.130:46659 every 8 connection(s)
I20260806 02:46:20.046638 15786 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
I20260806 02:46:20.055720 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 15786
I20260806 02:46:20.055915 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance
I20260806 02:46:20.056340 1758 external_mini_cluster.cc:1428] Setting key 221f0b227d87055bec4e7ff6993b5a5a
I20260806 02:46:20.056962 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:20.048740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:50039 (local address 127.1.183.190:40253)
0806 02:46:20.048895 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:20.048899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:20.050076 (+ 1177us) server_negotiation.cc:407] Connection header received
0806 02:46:20.051604 (+ 1528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:20.051608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:20.051674 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:20.051792 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:20.052857 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:20.053402 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:20.054616 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:20.054842 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:20.054960 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:20.056311 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:20.056314 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:20.056316 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:46:20.056332 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:20.056353 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:46:20.056361 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:20.056454 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:20.056731 (+ 277us) server_negotiation.cc:299] Negotiation successful
0806 02:46:20.056824 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 02:46:20.058142 15981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:20.049181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:40253 (local address 127.1.183.130:50039)
0806 02:46:20.049910 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:20.049956 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:20.051355 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:20.052004 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:20.052012 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:20.052108 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:20.052701 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:20.052715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:20.053548 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:20.053552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:20.054473 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:20.054484 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:20.055029 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:20.055034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:20.055204 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:20.055880 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:46:20.055882 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:20.055967 (+ 85us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:46:20.056083 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:20.056501 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:20.056514 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:20.056524 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 02:46:20.056664 (+ 140us) client_negotiation.cc:240] Negotiation successful
0806 02:46:20.057000 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":123,"threads_started":1}
I20260806 02:46:20.058892 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.131:0
--local_ip_for_outbound_sockets=127.1.183.131
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:40253
--builtin_ntp_servers=127.1.183.148:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 02:46:20.059581 15979 heartbeater.cc:347] Connected to a master server at 127.1.183.190:40253
I20260806 02:46:20.059870 15979 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:20.060527 15979 heartbeater.cc:511] Master 127.1.183.190:40253 requested a full tablet report, sending...
I20260806 02:46:20.061738 15505 ts_manager.cc:194] Registered new tserver with Master: 5a8518c14ea54ba0b7b615c8b404b2f3 (127.1.183.130:46659)
I20260806 02:46:20.062671 15505 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.183.130:50039
W20260806 02:46:20.181123 15982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:20.181459 15982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:20.181526 15982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:20.185675 15982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:20.185740 15982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:20.185796 15982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.131
I20260806 02:46:20.190937 15982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:40253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.131
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:46:20.192121 15982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:20.193656 15982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:20.204097 15992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:20.204335 15982 server_base.cc:1034] running on GCE node
W20260806 02:46:20.204453 15990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:20.204479 15989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:20.205013 15982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:20.205720 15982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:20.206928 15982 hybrid_clock.cc:648] HybridClock initialized: now 1785984380206916 us; error 35 us; skew 500 ppm
I20260806 02:46:20.209044 15982 webserver.cc:460] Webserver started at http://127.1.183.131:39737/ using document root <none> and password file <none>
I20260806 02:46:20.209682 15982 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:20.209754 15982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:20.209996 15982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:20.212319 15982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/data/instance:
uuid: "6a7724bd0c124691a7b0efa409fdb655"
format_stamp: "Formatted at 2026-08-06 02:46:20 on dist-test-slave-gmjp"
server_key: "7d89e77a140a1010ab5753a192c9fc9d"
server_key_iv: "0aa97cb8825bcce5b3cb8b9c53702156"
server_key_version: "encryptionkey@0"
I20260806 02:46:20.212962 15982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance:
uuid: "6a7724bd0c124691a7b0efa409fdb655"
format_stamp: "Formatted at 2026-08-06 02:46:20 on dist-test-slave-gmjp"
server_key: "7d89e77a140a1010ab5753a192c9fc9d"
server_key_iv: "0aa97cb8825bcce5b3cb8b9c53702156"
server_key_version: "encryptionkey@0"
I20260806 02:46:20.217581 15982 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:46:20.220615 16002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:20.222101 15982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:46:20.222239 15982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/wal
uuid: "6a7724bd0c124691a7b0efa409fdb655"
format_stamp: "Formatted at 2026-08-06 02:46:20 on dist-test-slave-gmjp"
server_key: "7d89e77a140a1010ab5753a192c9fc9d"
server_key_iv: "0aa97cb8825bcce5b3cb8b9c53702156"
server_key_version: "encryptionkey@0"
I20260806 02:46:20.222370 15982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:20.241932 15982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:20.242703 15982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:20.242929 15982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:20.243585 15982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:20.244658 15982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:20.244709 15982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:20.244774 15982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:20.244807 15982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:20.254060 15982 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.131:44177
I20260806 02:46:20.254091 16164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.131:44177 every 8 connection(s)
I20260806 02:46:20.255039 15982 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
I20260806 02:46:20.257923 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 15982
I20260806 02:46:20.258061 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance
I20260806 02:46:20.258411 1758 external_mini_cluster.cc:1428] Setting key 57a3cd503e203a3a817d798bb8e3d6b7
I20260806 02:46:20.265618 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:20.257102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:33795 (local address 127.1.183.190:40253)
0806 02:46:20.257273 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:20.257279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:20.258661 (+ 1382us) server_negotiation.cc:407] Connection header received
0806 02:46:20.260503 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:20.260507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:20.260565 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:20.260672 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:20.261635 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:20.262200 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:20.263621 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:20.263838 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:20.263919 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:20.265115 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:20.265119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:20.265121 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:46:20.265134 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:20.265153 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:46:20.265161 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:20.265260 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:20.265421 (+ 161us) server_negotiation.cc:299] Negotiation successful
0806 02:46:20.265487 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:46:20.266579 16170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:20.257606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:40253 (local address 127.1.183.131:33795)
0806 02:46:20.258427 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:20.258501 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:20.260267 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:20.260854 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:20.260862 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:20.260958 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:20.261461 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:20.261473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:20.262371 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:20.262374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:20.263493 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:20.263504 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:20.263977 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:20.263980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:20.264136 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:20.264838 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:46:20.264840 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:20.264884 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:46:20.264959 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:20.265261 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:20.265269 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:20.265274 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 02:46:20.265385 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 02:46:20.265635 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":158,"threads_started":1}
I20260806 02:46:20.267765 16166 heartbeater.cc:347] Connected to a master server at 127.1.183.190:40253
I20260806 02:46:20.268031 16166 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:20.268651 16166 heartbeater.cc:511] Master 127.1.183.190:40253 requested a full tablet report, sending...
I20260806 02:46:20.269765 15505 ts_manager.cc:194] Registered new tserver with Master: 6a7724bd0c124691a7b0efa409fdb655 (127.1.183.131:44177)
I20260806 02:46:20.270358 15505 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.183.131:33795
I20260806 02:46:20.272768 1758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 02:46:20.276686 15565 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:20.275686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46940 (local address 127.1.183.190:40253)
0806 02:46:20.275853 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:20.275860 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:20.276017 (+ 157us) server_negotiation.cc:407] Connection header received
0806 02:46:20.276103 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:20.276105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:20.276146 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:20.276260 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:20.276519 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46940: BlockingRecv error: recv got EOF from 127.0.0.1:46940 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 02:46:20.276785 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 15603
I20260806 02:46:20.295512 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 15786
I20260806 02:46:20.304214 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 15982
I20260806 02:46:20.317991 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 15460
2026-08-06T02:46:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (964 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-06T02:46:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:46:20Z Disabled control of system clock
I20260806 02:46:20.359081 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:33495
--webserver_interface=127.1.183.190
--webserver_port=0
--builtin_ntp_servers=127.1.183.148:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:33495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 02:46:20.479053 16184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:20.479379 16184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:20.479419 16184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:20.483597 16184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:46:20.483680 16184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:20.483696 16184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:20.483714 16184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:46:20.483736 16184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:46:20.488909 16184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:33495
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:33495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:46:20.490060 16184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:20.491482 16184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:20.500607 16195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:20.500710 16184 server_base.cc:1034] running on GCE node
W20260806 02:46:20.500607 16193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:20.500624 16198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:20.501591 16184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:20.502774 16184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:20.503953 16184 hybrid_clock.cc:648] HybridClock initialized: now 1785984380503939 us; error 37 us; skew 500 ppm
I20260806 02:46:20.506191 16184 webserver.cc:460] Webserver started at http://127.1.183.190:33781/ using document root <none> and password file <none>
I20260806 02:46:20.506829 16184 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:20.506901 16184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:20.507153 16184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:20.509492 16184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "57626856195f4ff58bece3ce9cc00b53"
format_stamp: "Formatted at 2026-08-06 02:46:20 on dist-test-slave-gmjp"
server_key: "126c9748617ebe72cf9c4419dd91eeee"
server_key_iv: "c670680e0f839f475b1f68ed2cb1c190"
server_key_version: "encryptionkey@0"
I20260806 02:46:20.510113 16184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "57626856195f4ff58bece3ce9cc00b53"
format_stamp: "Formatted at 2026-08-06 02:46:20 on dist-test-slave-gmjp"
server_key: "126c9748617ebe72cf9c4419dd91eeee"
server_key_iv: "c670680e0f839f475b1f68ed2cb1c190"
server_key_version: "encryptionkey@0"
I20260806 02:46:20.514226 16184 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:46:20.517285 16209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:20.518806 16184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:46:20.518991 16184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "57626856195f4ff58bece3ce9cc00b53"
format_stamp: "Formatted at 2026-08-06 02:46:20 on dist-test-slave-gmjp"
server_key: "126c9748617ebe72cf9c4419dd91eeee"
server_key_iv: "c670680e0f839f475b1f68ed2cb1c190"
server_key_version: "encryptionkey@0"
I20260806 02:46:20.519134 16184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:20.524212 16184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:20.524906 16184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:20.525115 16184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:20.532405 16184 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:33495
I20260806 02:46:20.532442 16295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:33495 every 8 connection(s)
I20260806 02:46:20.533484 16184 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:46:20.535925 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 16184
I20260806 02:46:20.536092 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:46:20.536423 1758 external_mini_cluster.cc:1428] Setting key 3846bd624b549458e5b66e33f7bbc4c4
I20260806 02:46:20.537045 16297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:20.542344 16301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:20.537856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47324 (local address 127.1.183.190:33495)
0806 02:46:20.538518 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:20.538534 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:20.538571 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:46:20.539234 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:20.539250 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:20.539608 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:20.539897 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:20.540962 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:20.540968 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:20.540982 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:46:20.540993 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:20.541051 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:46:20.541067 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:20.541205 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:20.541338 (+ 133us) server_negotiation.cc:299] Negotiation successful
0806 02:46:20.541630 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":144,"threads_started":1}
I20260806 02:46:20.544324 16297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57626856195f4ff58bece3ce9cc00b53: Bootstrap starting.
I20260806 02:46:20.546627 16297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57626856195f4ff58bece3ce9cc00b53: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:20.547348 16297 log.cc:824] T 00000000000000000000000000000000 P 57626856195f4ff58bece3ce9cc00b53: Log is configured to *not* fsync() on all Append() calls
I20260806 02:46:20.549575 16297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57626856195f4ff58bece3ce9cc00b53: No bootstrap required, opened a new log
I20260806 02:46:20.553536 16297 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57626856195f4ff58bece3ce9cc00b53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57626856195f4ff58bece3ce9cc00b53" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33495 } }
I20260806 02:46:20.553926 16297 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57626856195f4ff58bece3ce9cc00b53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:20.554025 16297 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57626856195f4ff58bece3ce9cc00b53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57626856195f4ff58bece3ce9cc00b53, State: Initialized, Role: FOLLOWER
I20260806 02:46:20.554582 16297 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57626856195f4ff58bece3ce9cc00b53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57626856195f4ff58bece3ce9cc00b53" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33495 } }
I20260806 02:46:20.554790 16297 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57626856195f4ff58bece3ce9cc00b53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:46:20.554859 16297 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57626856195f4ff58bece3ce9cc00b53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:46:20.554965 16297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57626856195f4ff58bece3ce9cc00b53 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:20.555980 16297 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57626856195f4ff58bece3ce9cc00b53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57626856195f4ff58bece3ce9cc00b53" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33495 } }
I20260806 02:46:20.556393 16297 leader_election.cc:302] T 00000000000000000000000000000000 P 57626856195f4ff58bece3ce9cc00b53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57626856195f4ff58bece3ce9cc00b53; no voters:
I20260806 02:46:20.556766 16297 leader_election.cc:288] T 00000000000000000000000000000000 P 57626856195f4ff58bece3ce9cc00b53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:46:20.557081 16303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57626856195f4ff58bece3ce9cc00b53 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:46:20.557348 16303 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57626856195f4ff58bece3ce9cc00b53 [term 1 LEADER]: Becoming Leader. State: Replica: 57626856195f4ff58bece3ce9cc00b53, State: Running, Role: LEADER
I20260806 02:46:20.557782 16303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57626856195f4ff58bece3ce9cc00b53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57626856195f4ff58bece3ce9cc00b53" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33495 } }
I20260806 02:46:20.558180 16297 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57626856195f4ff58bece3ce9cc00b53 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:46:20.559594 16305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57626856195f4ff58bece3ce9cc00b53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57626856195f4ff58bece3ce9cc00b53. Latest consensus state: current_term: 1 leader_uuid: "57626856195f4ff58bece3ce9cc00b53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57626856195f4ff58bece3ce9cc00b53" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33495 } } }
I20260806 02:46:20.559645 16304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57626856195f4ff58bece3ce9cc00b53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57626856195f4ff58bece3ce9cc00b53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57626856195f4ff58bece3ce9cc00b53" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33495 } } }
I20260806 02:46:20.559736 16304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57626856195f4ff58bece3ce9cc00b53 [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:20.559736 16305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57626856195f4ff58bece3ce9cc00b53 [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:20.560200 16316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:46:20.563527 16316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:46:20.569746 16316 catalog_manager.cc:1347] Generated new cluster ID: 2d4e80bb65104391b3d7f8994743b6e9
I20260806 02:46:20.569840 16316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:46:20.593770 16316 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:46:20.594194 16316 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:46:20.607363 16316 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57626856195f4ff58bece3ce9cc00b53: Generated new TSK 0
I20260806 02:46:20.608382 16316 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:46:20.613909 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.129:0
--local_ip_for_outbound_sockets=127.1.183.129
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:33495
--builtin_ntp_servers=127.1.183.148:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 02:46:20.745003 16328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:20.745503 16328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:20.745592 16328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:20.753175 16328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:20.753264 16328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:20.753418 16328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.129
I20260806 02:46:20.760675 16328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:33495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.129
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:46:20.761919 16328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:20.763556 16328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:20.775032 16334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:20.775121 16328 server_base.cc:1034] running on GCE node
W20260806 02:46:20.775046 16333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:20.776623 16338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:20.777137 16328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:20.777885 16328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:20.779054 16328 hybrid_clock.cc:648] HybridClock initialized: now 1785984380779023 us; error 53 us; skew 500 ppm
I20260806 02:46:20.781044 16328 webserver.cc:460] Webserver started at http://127.1.183.129:42811/ using document root <none> and password file <none>
I20260806 02:46:20.781615 16328 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:20.781678 16328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:20.781921 16328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:20.784116 16328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/data/instance:
uuid: "c1865c49ecc34d66b8761141edee005e"
format_stamp: "Formatted at 2026-08-06 02:46:20 on dist-test-slave-gmjp"
server_key: "50e93b26cd0ac6701bcdc6ad005cd6b4"
server_key_iv: "30f1dd5de746dace487bb1a0f5fe89f1"
server_key_version: "encryptionkey@0"
I20260806 02:46:20.785012 16328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance:
uuid: "c1865c49ecc34d66b8761141edee005e"
format_stamp: "Formatted at 2026-08-06 02:46:20 on dist-test-slave-gmjp"
server_key: "50e93b26cd0ac6701bcdc6ad005cd6b4"
server_key_iv: "30f1dd5de746dace487bb1a0f5fe89f1"
server_key_version: "encryptionkey@0"
I20260806 02:46:20.790508 16328 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:46:20.793421 16347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:20.794889 16328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:46:20.795003 16328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/wal
uuid: "c1865c49ecc34d66b8761141edee005e"
format_stamp: "Formatted at 2026-08-06 02:46:20 on dist-test-slave-gmjp"
server_key: "50e93b26cd0ac6701bcdc6ad005cd6b4"
server_key_iv: "30f1dd5de746dace487bb1a0f5fe89f1"
server_key_version: "encryptionkey@0"
I20260806 02:46:20.795192 16328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:20.800860 16328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:20.801568 16328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:20.801769 16328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:20.802361 16328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:20.803335 16328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:20.803381 16328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:20.803458 16328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:20.803490 16328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:20.813138 16328 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.129:44947
I20260806 02:46:20.813169 16502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.129:44947 every 8 connection(s)
I20260806 02:46:20.814154 16328 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
I20260806 02:46:20.816319 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 16328
I20260806 02:46:20.816473 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance
I20260806 02:46:20.816772 1758 external_mini_cluster.cc:1428] Setting key 7ac3110ce720ec5a31e7ec872a76fc9e
I20260806 02:46:20.819128 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.130:0
--local_ip_for_outbound_sockets=127.1.183.130
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:33495
--builtin_ntp_servers=127.1.183.148:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 02:46:20.821795 16301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:20.816422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:40625 (local address 127.1.183.190:33495)
0806 02:46:20.816582 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:20.816587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:20.818186 (+ 1599us) server_negotiation.cc:407] Connection header received
0806 02:46:20.819760 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:20.819774 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:20.819828 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:20.819931 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:20.821267 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:20.821271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:20.821274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:46:20.821290 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:20.821316 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:46:20.821340 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:20.821444 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:20.821600 (+ 156us) server_negotiation.cc:299] Negotiation successful
0806 02:46:20.821652 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 02:46:20.822841 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:20.817028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:33495 (local address 127.1.183.129:40625)
0806 02:46:20.817951 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:20.818014 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:20.819520 (+ 1506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:20.820107 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:20.820114 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:20.820312 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:20.820982 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:46:20.820983 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:20.821029 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:46:20.821106 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:20.821436 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:20.821447 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:20.821456 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 02:46:20.821653 (+ 197us) client_negotiation.cc:240] Negotiation successful
0806 02:46:20.821973 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":149,"threads_started":1}
I20260806 02:46:20.824476 16504 heartbeater.cc:347] Connected to a master server at 127.1.183.190:33495
I20260806 02:46:20.824764 16504 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:20.825238 16504 heartbeater.cc:511] Master 127.1.183.190:33495 requested a full tablet report, sending...
I20260806 02:46:20.827242 16232 ts_manager.cc:194] Registered new tserver with Master: c1865c49ecc34d66b8761141edee005e (127.1.183.129:44947)
W20260806 02:46:20.939368 16507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:20.939659 16507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:20.939702 16507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:20.944135 16507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:20.944226 16507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:20.944289 16507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.130
I20260806 02:46:20.949070 16507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:33495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.130
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:46:20.950196 16507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:20.951609 16507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:20.962862 16516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:20.963176 16513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:20.964501 16507 server_base.cc:1034] running on GCE node
W20260806 02:46:20.965044 16512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:20.965693 16507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:20.966584 16507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:20.967795 16507 hybrid_clock.cc:648] HybridClock initialized: now 1785984380967769 us; error 38 us; skew 500 ppm
I20260806 02:46:20.969959 16507 webserver.cc:460] Webserver started at http://127.1.183.130:34551/ using document root <none> and password file <none>
I20260806 02:46:20.970638 16507 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:20.970705 16507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:20.970952 16507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:20.973300 16507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/data/instance:
uuid: "0f0b03dc40a14b90ac7266e742a49402"
format_stamp: "Formatted at 2026-08-06 02:46:20 on dist-test-slave-gmjp"
server_key: "f87469c9ad782a3db270458337fc81a9"
server_key_iv: "20158cd5a4bd424c555b7ccb6d4a545a"
server_key_version: "encryptionkey@0"
I20260806 02:46:20.973948 16507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance:
uuid: "0f0b03dc40a14b90ac7266e742a49402"
format_stamp: "Formatted at 2026-08-06 02:46:20 on dist-test-slave-gmjp"
server_key: "f87469c9ad782a3db270458337fc81a9"
server_key_iv: "20158cd5a4bd424c555b7ccb6d4a545a"
server_key_version: "encryptionkey@0"
I20260806 02:46:20.978217 16507 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:46:20.981195 16525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:20.982836 16507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:46:20.982991 16507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/wal
uuid: "0f0b03dc40a14b90ac7266e742a49402"
format_stamp: "Formatted at 2026-08-06 02:46:20 on dist-test-slave-gmjp"
server_key: "f87469c9ad782a3db270458337fc81a9"
server_key_iv: "20158cd5a4bd424c555b7ccb6d4a545a"
server_key_version: "encryptionkey@0"
I20260806 02:46:20.983144 16507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:20.988327 16507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:20.989035 16507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:20.989245 16507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:20.989862 16507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:20.990859 16507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:20.990908 16507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:20.990972 16507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:20.991010 16507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:21.000734 16507 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.130:34185
I20260806 02:46:21.000748 16689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.130:34185 every 8 connection(s)
I20260806 02:46:21.001782 16507 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
I20260806 02:46:21.008004 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 16507
I20260806 02:46:21.008152 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance
I20260806 02:46:21.008479 16301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:21.003873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:58911 (local address 127.1.183.190:33495)
0806 02:46:21.004041 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:21.004045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:21.005167 (+ 1122us) server_negotiation.cc:407] Connection header received
0806 02:46:21.006572 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:21.006575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:21.006622 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:21.006743 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:21.007848 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:21.007852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:21.007855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:46:21.007875 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:21.007893 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:46:21.007901 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:21.008002 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:21.008252 (+ 250us) server_negotiation.cc:299] Negotiation successful
0806 02:46:21.008311 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 02:46:21.008544 1758 external_mini_cluster.cc:1428] Setting key d25e43e387520017985a6fa91dd6ab83
I20260806 02:46:21.009874 16692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:21.004303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:33495 (local address 127.1.183.130:58911)
0806 02:46:21.005009 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:21.005053 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:21.006346 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:21.006917 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:21.006924 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:21.007014 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:21.007563 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:46:21.007564 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:21.007605 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:46:21.007676 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:21.007994 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:21.008004 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:21.008011 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 02:46:21.008736 (+ 725us) client_negotiation.cc:240] Negotiation successful
0806 02:46:21.008952 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":120,"threads_started":1}
I20260806 02:46:21.010911 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.131:0
--local_ip_for_outbound_sockets=127.1.183.131
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:33495
--builtin_ntp_servers=127.1.183.148:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 02:46:21.012239 16690 heartbeater.cc:347] Connected to a master server at 127.1.183.190:33495
I20260806 02:46:21.012526 16690 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:21.013039 16690 heartbeater.cc:511] Master 127.1.183.190:33495 requested a full tablet report, sending...
I20260806 02:46:21.014286 16232 ts_manager.cc:194] Registered new tserver with Master: 0f0b03dc40a14b90ac7266e742a49402 (127.1.183.130:34185)
W20260806 02:46:21.131834 16696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:21.132136 16696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:21.132293 16696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:21.136521 16696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:21.136590 16696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:21.136649 16696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.131
I20260806 02:46:21.141736 16696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:33495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.131
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:46:21.142967 16696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:21.144493 16696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:21.155017 16704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:21.155017 16701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:21.155073 16702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:21.155409 16696 server_base.cc:1034] running on GCE node
I20260806 02:46:21.155841 16696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:21.156599 16696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:21.157789 16696 hybrid_clock.cc:648] HybridClock initialized: now 1785984381157779 us; error 45 us; skew 500 ppm
I20260806 02:46:21.160116 16696 webserver.cc:460] Webserver started at http://127.1.183.131:43013/ using document root <none> and password file <none>
I20260806 02:46:21.160816 16696 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:21.160885 16696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:21.161075 16696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:21.163200 16696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/data/instance:
uuid: "5eaff6f8d1b24107addc7b8e031d40f6"
format_stamp: "Formatted at 2026-08-06 02:46:21 on dist-test-slave-gmjp"
server_key: "a4d2904642475b6eaa99859d904b8044"
server_key_iv: "fd0814d05edc232f6461e193153cb88e"
server_key_version: "encryptionkey@0"
I20260806 02:46:21.163743 16696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance:
uuid: "5eaff6f8d1b24107addc7b8e031d40f6"
format_stamp: "Formatted at 2026-08-06 02:46:21 on dist-test-slave-gmjp"
server_key: "a4d2904642475b6eaa99859d904b8044"
server_key_iv: "fd0814d05edc232f6461e193153cb88e"
server_key_version: "encryptionkey@0"
I20260806 02:46:21.167614 16696 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260806 02:46:21.170385 16713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:21.171829 16696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:46:21.171978 16696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/wal
uuid: "5eaff6f8d1b24107addc7b8e031d40f6"
format_stamp: "Formatted at 2026-08-06 02:46:21 on dist-test-slave-gmjp"
server_key: "a4d2904642475b6eaa99859d904b8044"
server_key_iv: "fd0814d05edc232f6461e193153cb88e"
server_key_version: "encryptionkey@0"
I20260806 02:46:21.172106 16696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:21.177110 16696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:21.177783 16696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:21.177994 16696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:21.178602 16696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:21.179572 16696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:21.179656 16696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:21.179744 16696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:21.179796 16696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:21.192289 16696 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.131:37429
I20260806 02:46:21.192333 16894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.131:37429 every 8 connection(s)
I20260806 02:46:21.193518 16696 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
I20260806 02:46:21.199463 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 16696
I20260806 02:46:21.199599 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance
I20260806 02:46:21.199954 1758 external_mini_cluster.cc:1428] Setting key 8ef8ba6c686d714480b3afb7ba61aa6e
I20260806 02:46:21.200740 16301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:21.195547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:43727 (local address 127.1.183.190:33495)
0806 02:46:21.195705 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:21.195711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:21.196911 (+ 1200us) server_negotiation.cc:407] Connection header received
0806 02:46:21.198376 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:21.198381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:21.198486 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:21.198630 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:21.200143 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:21.200147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:21.200181 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:46:21.200199 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:21.200223 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:46:21.200236 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:21.200359 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:21.200532 (+ 173us) server_negotiation.cc:299] Negotiation successful
0806 02:46:21.200605 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:46:21.202076 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:21.195875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:33495 (local address 127.1.183.131:43727)
0806 02:46:21.196742 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:21.196787 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:21.198131 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:21.198849 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:21.198860 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:21.198983 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:21.199700 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:46:21.199748 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:21.199812 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:46:21.199892 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:21.200353 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:21.200362 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:21.200370 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 02:46:21.200569 (+ 199us) client_negotiation.cc:240] Negotiation successful
0806 02:46:21.200950 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":200,"threads_started":1}
I20260806 02:46:21.203215 16899 heartbeater.cc:347] Connected to a master server at 127.1.183.190:33495
I20260806 02:46:21.203481 16899 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:21.203969 16899 heartbeater.cc:511] Master 127.1.183.190:33495 requested a full tablet report, sending...
I20260806 02:46:21.205180 16232 ts_manager.cc:194] Registered new tserver with Master: 5eaff6f8d1b24107addc7b8e031d40f6 (127.1.183.131:37429)
I20260806 02:46:21.214582 1758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 02:46:21.217778 16301 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:46:21.216790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47326 (local address 127.1.183.190:33495)
0806 02:46:21.216949 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:21.216954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:21.217142 (+ 188us) server_negotiation.cc:407] Connection header received
0806 02:46:21.217236 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:21.217238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:21.217274 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:21.217344 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:21.217616 (+ 272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47326: BlockingRecv error: recv got EOF from 127.0.0.1:47326 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:46:21.217916 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 16328
I20260806 02:46:21.239137 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 16507
I20260806 02:46:21.251511 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 16696
I20260806 02:46:21.264293 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 16184
2026-08-06T02:46:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (943 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:46:21 dist-test-slave-gmjp krb5kdc[16914](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:46:21 dist-test-slave-gmjp krb5kdc[16914](info): set up 2 sockets
Aug 06 02:46:21 dist-test-slave-gmjp krb5kdc[16914](info): commencing operation
krb5kdc: starting...
W20260806 02:46:22.510561 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.218s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:46:22 dist-test-slave-gmjp krb5kdc[16914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:46:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:46:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
I20260806 02:46:22.651422 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:44353
--webserver_interface=127.1.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:35991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:44353
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:46:22.767599 16935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:22.767958 16935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:22.768054 16935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:22.772245 16935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:46:22.772365 16935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:22.772403 16935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:22.772445 16935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:46:22.772480 16935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:46:22.777979 16935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:35991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:44353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:44353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:46:22.779232 16935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:22.780368 16935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:22.789460 16948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:22.789508 16945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:22.789489 16935 server_base.cc:1034] running on GCE node
W20260806 02:46:22.789474 16943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:22.790334 16935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:22.791509 16935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:22.792675 16935 hybrid_clock.cc:648] HybridClock initialized: now 1785984382792661 us; error 25 us; skew 500 ppm
Aug 06 02:46:22 dist-test-slave-gmjp krb5kdc[16914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984382, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:22.795586 16935 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:46:22.796617 16935 webserver.cc:460] Webserver started at http://127.1.183.190:38911/ using document root <none> and password file <none>
I20260806 02:46:22.797173 16935 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:22.797235 16935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:22.797417 16935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:22.799538 16935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "222237f7a26f401f903c622b1ea9af30"
format_stamp: "Formatted at 2026-08-06 02:46:22 on dist-test-slave-gmjp"
server_key: "ef24f805d121ac4bca233a89e5ae3947"
server_key_iv: "05f8722bf75efeceb43478af5b18e864"
server_key_version: "encryptionkey@0"
I20260806 02:46:22.800243 16935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "222237f7a26f401f903c622b1ea9af30"
format_stamp: "Formatted at 2026-08-06 02:46:22 on dist-test-slave-gmjp"
server_key: "ef24f805d121ac4bca233a89e5ae3947"
server_key_iv: "05f8722bf75efeceb43478af5b18e864"
server_key_version: "encryptionkey@0"
I20260806 02:46:22.804124 16935 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 02:46:22.806799 16959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:22.808287 16935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:46:22.808434 16935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "222237f7a26f401f903c622b1ea9af30"
format_stamp: "Formatted at 2026-08-06 02:46:22 on dist-test-slave-gmjp"
server_key: "ef24f805d121ac4bca233a89e5ae3947"
server_key_iv: "05f8722bf75efeceb43478af5b18e864"
server_key_version: "encryptionkey@0"
I20260806 02:46:22.808555 16935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:22.825646 16935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:22.828678 16935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:22.828896 16935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:22.836107 16935 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:44353
I20260806 02:46:22.836437 17033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:44353 every 8 connection(s)
I20260806 02:46:22.837157 16935 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:46:22.839706 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 16935
I20260806 02:46:22.839874 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:46:22.840143 17034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:22.840199 1758 external_mini_cluster.cc:1428] Setting key c50ed22ffb0b8661e00910a3cf84136d
I20260806 02:46:22.846678 17034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 222237f7a26f401f903c622b1ea9af30: Bootstrap starting.
I20260806 02:46:22.849232 17034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 222237f7a26f401f903c622b1ea9af30: Neither blocks nor log segments found. Creating new log.
Aug 06 02:46:22 dist-test-slave-gmjp krb5kdc[16914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:46:22.850023 17034 log.cc:824] T 00000000000000000000000000000000 P 222237f7a26f401f903c622b1ea9af30: Log is configured to *not* fsync() on all Append() calls
I20260806 02:46:22.852234 17034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 222237f7a26f401f903c622b1ea9af30: No bootstrap required, opened a new log
I20260806 02:46:22.855366 17038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:22.841701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51884 (local address 127.1.183.190:44353)
0806 02:46:22.842173 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:22.842188 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:22.842225 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:46:22.843005 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:22.843024 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:22.843349 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:22.843687 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:22.844563 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:22.845568 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:22.846452 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:22.846994 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:22.847181 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:22.849928 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:22.849934 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:22.849949 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:22.849979 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:22.851882 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:22.852467 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:22.852474 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:22.852480 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:22.852633 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:22.852934 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:22.852937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:22.852939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:22.853459 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:22.853656 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:22.853923 (+ 267us) server_negotiation.cc:299] Negotiation successful
0806 02:46:22.854244 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":145,"threads_started":1}
I20260806 02:46:22.856911 17034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 222237f7a26f401f903c622b1ea9af30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "222237f7a26f401f903c622b1ea9af30" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 44353 } }
I20260806 02:46:22.857134 17034 raft_consensus.cc:374] T 00000000000000000000000000000000 P 222237f7a26f401f903c622b1ea9af30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:22.857205 17034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 222237f7a26f401f903c622b1ea9af30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 222237f7a26f401f903c622b1ea9af30, State: Initialized, Role: FOLLOWER
I20260806 02:46:22.857633 17034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 222237f7a26f401f903c622b1ea9af30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "222237f7a26f401f903c622b1ea9af30" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 44353 } }
I20260806 02:46:22.857803 17034 raft_consensus.cc:388] T 00000000000000000000000000000000 P 222237f7a26f401f903c622b1ea9af30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:46:22.857877 17034 raft_consensus.cc:482] T 00000000000000000000000000000000 P 222237f7a26f401f903c622b1ea9af30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:46:22.858065 17034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 222237f7a26f401f903c622b1ea9af30 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:22.859481 17034 raft_consensus.cc:504] T 00000000000000000000000000000000 P 222237f7a26f401f903c622b1ea9af30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "222237f7a26f401f903c622b1ea9af30" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 44353 } }
I20260806 02:46:22.859792 17034 leader_election.cc:302] T 00000000000000000000000000000000 P 222237f7a26f401f903c622b1ea9af30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 222237f7a26f401f903c622b1ea9af30; no voters:
I20260806 02:46:22.860117 17034 leader_election.cc:288] T 00000000000000000000000000000000 P 222237f7a26f401f903c622b1ea9af30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:46:22.860220 17040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 222237f7a26f401f903c622b1ea9af30 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:46:22.860531 17040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 222237f7a26f401f903c622b1ea9af30 [term 1 LEADER]: Becoming Leader. State: Replica: 222237f7a26f401f903c622b1ea9af30, State: Running, Role: LEADER
I20260806 02:46:22.860852 17040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 222237f7a26f401f903c622b1ea9af30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "222237f7a26f401f903c622b1ea9af30" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 44353 } }
I20260806 02:46:22.861552 17034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 222237f7a26f401f903c622b1ea9af30 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:46:22.862864 17043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 222237f7a26f401f903c622b1ea9af30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 222237f7a26f401f903c622b1ea9af30. Latest consensus state: current_term: 1 leader_uuid: "222237f7a26f401f903c622b1ea9af30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "222237f7a26f401f903c622b1ea9af30" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 44353 } } }
I20260806 02:46:22.862870 17041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 222237f7a26f401f903c622b1ea9af30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "222237f7a26f401f903c622b1ea9af30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "222237f7a26f401f903c622b1ea9af30" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 44353 } } }
I20260806 02:46:22.863106 17043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 222237f7a26f401f903c622b1ea9af30 [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:22.863138 17041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 222237f7a26f401f903c622b1ea9af30 [sys.catalog]: This master's current role is: LEADER
I20260806 02:46:22.864130 17056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:46:22.866806 17056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:46:22.872838 17056 catalog_manager.cc:1347] Generated new cluster ID: af405937b72a4b84bff6467dfc7eea5b
I20260806 02:46:22.872922 17056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:46:22.882658 17056 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:46:22.883940 17056 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:46:22.889866 17056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 222237f7a26f401f903c622b1ea9af30: Generated new TSK 0
I20260806 02:46:22.890568 17056 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
I20260806 02:46:22.931641 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.129:0
--local_ip_for_outbound_sockets=127.1.183.129
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:44353
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:35991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:46:23.046299 17071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:23.046559 17071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:23.046602 17071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:23.050482 17071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:23.050542 17071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:23.050598 17071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.129
I20260806 02:46:23.055786 17071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:35991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:44353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.129
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:46:23.056995 17071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:23.058014 17071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:23.068075 17079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:23.068087 17077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:23.068202 17081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:23.068325 17071 server_base.cc:1034] running on GCE node
I20260806 02:46:23.068913 17071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:23.069587 17071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:23.070791 17071 hybrid_clock.cc:648] HybridClock initialized: now 1785984383070767 us; error 32 us; skew 500 ppm
Aug 06 02:46:23 dist-test-slave-gmjp krb5kdc[16914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984383, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:23.077906 17071 init.cc:377] Logged in from keytab as kudu/127.1.183.129@KRBTEST.COM (short username kudu)
I20260806 02:46:23.078965 17071 webserver.cc:460] Webserver started at http://127.1.183.129:39443/ using document root <none> and password file <none>
I20260806 02:46:23.079514 17071 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:23.079576 17071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:23.079757 17071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:23.081879 17071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/data/instance:
uuid: "724b769ff3624f7c892469118e45120e"
format_stamp: "Formatted at 2026-08-06 02:46:23 on dist-test-slave-gmjp"
server_key: "bdd1edf64eaebb24e22be73a80095e33"
server_key_iv: "7513a48484af5b271532be0ee53671f8"
server_key_version: "encryptionkey@0"
I20260806 02:46:23.082409 17071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance:
uuid: "724b769ff3624f7c892469118e45120e"
format_stamp: "Formatted at 2026-08-06 02:46:23 on dist-test-slave-gmjp"
server_key: "bdd1edf64eaebb24e22be73a80095e33"
server_key_iv: "7513a48484af5b271532be0ee53671f8"
server_key_version: "encryptionkey@0"
I20260806 02:46:23.086278 17071 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260806 02:46:23.088737 17091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:23.090054 17071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:46:23.090224 17071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/wal
uuid: "724b769ff3624f7c892469118e45120e"
format_stamp: "Formatted at 2026-08-06 02:46:23 on dist-test-slave-gmjp"
server_key: "bdd1edf64eaebb24e22be73a80095e33"
server_key_iv: "7513a48484af5b271532be0ee53671f8"
server_key_version: "encryptionkey@0"
I20260806 02:46:23.090349 17071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:23.123888 17071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:23.127236 17071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:23.127488 17071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:23.128217 17071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:23.129180 17071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:23.129261 17071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:23.129339 17071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:23.129386 17071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:23.138886 17071 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.129:44147
I20260806 02:46:23.138911 17255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.129:44147 every 8 connection(s)
I20260806 02:46:23.139860 17071 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
Aug 06 02:46:23 dist-test-slave-gmjp krb5kdc[16914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984383, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:46:23.149641 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 17071
I20260806 02:46:23.149791 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance
I20260806 02:46:23.150112 1758 external_mini_cluster.cc:1428] Setting key 97fbc7dc6484910ec801cd10aa237419
I20260806 02:46:23.154400 17038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:23.141833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:38349 (local address 127.1.183.190:44353)
0806 02:46:23.141989 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:23.141993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:23.142853 (+ 860us) server_negotiation.cc:407] Connection header received
0806 02:46:23.144310 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:23.144313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:23.144361 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:23.144448 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:23.145659 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.146207 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.147363 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.147607 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.147686 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:23.150333 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:23.150337 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:23.150341 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:23.150370 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:23.152141 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:23.152677 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:23.152683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:23.152688 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:23.152787 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:23.153151 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:23.153157 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:23.153161 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:23.153421 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:23.153531 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:23.154092 (+ 561us) server_negotiation.cc:299] Negotiation successful
0806 02:46:23.154231 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:46:23.155316 17258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:23.142217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:44353 (local address 127.1.183.129:38349)
0806 02:46:23.142682 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:23.142728 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:23.144019 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:23.144610 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:23.144616 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:23.145022 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:23.145516 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:23.145529 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.146325 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.146328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:23.147244 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:23.147253 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.147737 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.147740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:23.147915 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:23.148575 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:23.148592 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:23.150135 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:23.152281 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:23.152286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:23.152299 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:23.152575 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:23.152893 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:23.152895 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:23.152896 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:23.153059 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:23.153535 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:23.153540 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:23.153788 (+ 248us) client_negotiation.cc:769] Sending connection context
0806 02:46:23.154047 (+ 259us) client_negotiation.cc:240] Negotiation successful
0806 02:46:23.154393 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":114,"threads_started":1}
I20260806 02:46:23.156728 17256 heartbeater.cc:347] Connected to a master server at 127.1.183.190:44353
I20260806 02:46:23.157023 17256 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:23.157737 17256 heartbeater.cc:511] Master 127.1.183.190:44353 requested a full tablet report, sending...
I20260806 02:46:23.159770 16980 ts_manager.cc:194] Registered new tserver with Master: 724b769ff3624f7c892469118e45120e (127.1.183.129:44147)
I20260806 02:46:23.161569 16980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.129@KRBTEST.COM'} at 127.1.183.129:38349
WARNING: no policy specified for kudu/127.1.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
I20260806 02:46:23.182928 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.130:0
--local_ip_for_outbound_sockets=127.1.183.130
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:44353
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:35991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:46:23.304409 17262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:23.304706 17262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:23.304751 17262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:23.309010 17262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:23.309082 17262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:23.309145 17262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.130
I20260806 02:46:23.314841 17262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:35991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:44353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.130
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:46:23.316009 17262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:23.317123 17262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:23.327513 17272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:23.327607 17271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:23.327515 17274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:23.327775 17262 server_base.cc:1034] running on GCE node
I20260806 02:46:23.328357 17262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:23.329106 17262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:23.330308 17262 hybrid_clock.cc:648] HybridClock initialized: now 1785984383330298 us; error 32 us; skew 500 ppm
Aug 06 02:46:23 dist-test-slave-gmjp krb5kdc[16914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984383, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:23.333655 17262 init.cc:377] Logged in from keytab as kudu/127.1.183.130@KRBTEST.COM (short username kudu)
I20260806 02:46:23.334721 17262 webserver.cc:460] Webserver started at http://127.1.183.130:34257/ using document root <none> and password file <none>
I20260806 02:46:23.335376 17262 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:23.335446 17262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:23.335709 17262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:23.338143 17262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/data/instance:
uuid: "5eed8ed6b0584be3968dbb6263753502"
format_stamp: "Formatted at 2026-08-06 02:46:23 on dist-test-slave-gmjp"
server_key: "430ee1652a32423930b6b0de6e226356"
server_key_iv: "d5d5523d56051f34c5913a186eb13cdc"
server_key_version: "encryptionkey@0"
I20260806 02:46:23.338843 17262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance:
uuid: "5eed8ed6b0584be3968dbb6263753502"
format_stamp: "Formatted at 2026-08-06 02:46:23 on dist-test-slave-gmjp"
server_key: "430ee1652a32423930b6b0de6e226356"
server_key_iv: "d5d5523d56051f34c5913a186eb13cdc"
server_key_version: "encryptionkey@0"
I20260806 02:46:23.343355 17262 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260806 02:46:23.346334 17284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:23.347831 17262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:46:23.348011 17262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/wal
uuid: "5eed8ed6b0584be3968dbb6263753502"
format_stamp: "Formatted at 2026-08-06 02:46:23 on dist-test-slave-gmjp"
server_key: "430ee1652a32423930b6b0de6e226356"
server_key_iv: "d5d5523d56051f34c5913a186eb13cdc"
server_key_version: "encryptionkey@0"
I20260806 02:46:23.348201 17262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:23.362531 17262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:23.366139 17262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:23.366330 17262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:23.366909 17262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:23.367746 17262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:23.367796 17262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:23.367837 17262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:23.367852 17262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:23.377194 17262 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.130:38195
I20260806 02:46:23.377262 17472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.130:38195 every 8 connection(s)
I20260806 02:46:23.378199 17262 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
I20260806 02:46:23.383982 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 17262
I20260806 02:46:23.384188 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance
I20260806 02:46:23.384495 1758 external_mini_cluster.cc:1428] Setting key 6924cb4f001868131a9c9af44408497c
Aug 06 02:46:23 dist-test-slave-gmjp krb5kdc[16914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984383, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:46:23.395081 17038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:23.380317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:55937 (local address 127.1.183.190:44353)
0806 02:46:23.380469 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:23.380473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:23.381361 (+ 888us) server_negotiation.cc:407] Connection header received
0806 02:46:23.382788 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:23.382793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:23.382856 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:23.382932 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:23.384234 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.384870 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.386691 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.386941 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.387067 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:23.390353 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:23.390360 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:23.390366 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:23.390411 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:23.392527 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:23.393164 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:23.393170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:23.393173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:23.393285 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:23.393654 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:23.393659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:23.393663 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:23.393996 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:23.394110 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:23.394711 (+ 601us) server_negotiation.cc:299] Negotiation successful
0806 02:46:23.394866 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:46:23.396059 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:23.380698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:44353 (local address 127.1.183.130:55937)
0806 02:46:23.381182 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:23.381232 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:23.382559 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:23.383110 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:23.383117 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:23.383507 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:23.384032 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:23.384046 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.385012 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.385016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:23.386547 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:23.386559 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.387120 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.387124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:23.387310 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:23.387988 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:23.388007 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:23.390144 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:23.392663 (+ 2519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:23.392670 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:23.392685 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:23.393034 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:23.393387 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:23.393389 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:23.393391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:23.393547 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:23.394125 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:23.394131 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:23.394433 (+ 302us) client_negotiation.cc:769] Sending connection context
0806 02:46:23.394616 (+ 183us) client_negotiation.cc:240] Negotiation successful
0806 02:46:23.395072 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":126,"threads_started":1}
WARNING: no policy specified for kudu/127.1.183.131@KRBTEST.COM; defaulting to no policy
I20260806 02:46:23.397493 17474 heartbeater.cc:347] Connected to a master server at 127.1.183.190:44353
I20260806 02:46:23.397791 17474 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:23.398350 17474 heartbeater.cc:511] Master 127.1.183.190:44353 requested a full tablet report, sending...
I20260806 02:46:23.399634 16980 ts_manager.cc:194] Registered new tserver with Master: 5eed8ed6b0584be3968dbb6263753502 (127.1.183.130:38195)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.131@KRBTEST.COM" created.
I20260806 02:46:23.400485 16980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.130@KRBTEST.COM'} at 127.1.183.130:55937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
I20260806 02:46:23.419018 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.131:0
--local_ip_for_outbound_sockets=127.1.183.131
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:44353
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:35991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:46:23.539800 17481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:46:23.540129 17481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:46:23.540251 17481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:46:23.544528 17481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:46:23.544600 17481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:46:23.544693 17481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.131
I20260806 02:46:23.550242 17481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:35991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:44353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.131
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:46:23.551445 17481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:46:23.552682 17481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:46:23.563033 17488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:23.563040 17486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:46:23.563061 17490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:46:23.563441 17481 server_base.cc:1034] running on GCE node
I20260806 02:46:23.563879 17481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:46:23.564643 17481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:46:23.565837 17481 hybrid_clock.cc:648] HybridClock initialized: now 1785984383565806 us; error 56 us; skew 500 ppm
Aug 06 02:46:23 dist-test-slave-gmjp krb5kdc[16914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984383, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:46:23.569001 17481 init.cc:377] Logged in from keytab as kudu/127.1.183.131@KRBTEST.COM (short username kudu)
I20260806 02:46:23.570004 17481 webserver.cc:460] Webserver started at http://127.1.183.131:35585/ using document root <none> and password file <none>
I20260806 02:46:23.570660 17481 fs_manager.cc:359] Metadata directory not provided
I20260806 02:46:23.570775 17481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:46:23.571040 17481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:46:23.573447 17481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/data/instance:
uuid: "563bce46a5bb469491139329ee43e544"
format_stamp: "Formatted at 2026-08-06 02:46:23 on dist-test-slave-gmjp"
server_key: "f1171fcbdc452e5275eab4bf30ddfcc9"
server_key_iv: "408fe056e52cb9bbf094af5204c50363"
server_key_version: "encryptionkey@0"
I20260806 02:46:23.574126 17481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance:
uuid: "563bce46a5bb469491139329ee43e544"
format_stamp: "Formatted at 2026-08-06 02:46:23 on dist-test-slave-gmjp"
server_key: "f1171fcbdc452e5275eab4bf30ddfcc9"
server_key_iv: "408fe056e52cb9bbf094af5204c50363"
server_key_version: "encryptionkey@0"
I20260806 02:46:23.578357 17481 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:46:23.580946 17497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:23.582312 17481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:46:23.582463 17481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/wal
uuid: "563bce46a5bb469491139329ee43e544"
format_stamp: "Formatted at 2026-08-06 02:46:23 on dist-test-slave-gmjp"
server_key: "f1171fcbdc452e5275eab4bf30ddfcc9"
server_key_iv: "408fe056e52cb9bbf094af5204c50363"
server_key_version: "encryptionkey@0"
I20260806 02:46:23.582580 17481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:46:23.599128 17481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:46:23.602969 17481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:46:23.603199 17481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:46:23.603919 17481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:46:23.604980 17481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:46:23.605037 17481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:23.605079 17481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:46:23.605103 17481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:46:23.614400 17481 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.131:37719
I20260806 02:46:23.614421 17680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.131:37719 every 8 connection(s)
I20260806 02:46:23.615365 17481 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
I20260806 02:46:23.617090 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 17481
I20260806 02:46:23.617244 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance
I20260806 02:46:23.617563 1758 external_mini_cluster.cc:1428] Setting key db3d35e1f66f04785fc09e951af7d6e3
Aug 06 02:46:23 dist-test-slave-gmjp krb5kdc[16914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984383, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:46:23.631101 17038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:23.617974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:56879 (local address 127.1.183.190:44353)
0806 02:46:23.618132 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:23.618137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:23.618622 (+ 485us) server_negotiation.cc:407] Connection header received
0806 02:46:23.620389 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:23.620393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:23.620454 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:23.620559 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:23.621810 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.622364 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.623578 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.623815 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.623924 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:23.626886 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:23.626890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:23.626892 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:23.626942 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:23.628516 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:23.629241 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:23.629247 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:23.629249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:23.629348 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:23.629707 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:23.629709 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:23.629710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:23.629922 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:23.630057 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:23.630747 (+ 690us) server_negotiation.cc:299] Negotiation successful
0806 02:46:23.630884 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:46:23.632551 17683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:23.617862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:44353 (local address 127.1.183.131:56879)
0806 02:46:23.618471 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:23.618536 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:23.620114 (+ 1578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:23.620754 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:23.620762 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:23.621149 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:46:23.621654 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:23.621667 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.622502 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.622506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:23.623437 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:23.623448 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.623991 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.623995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:23.624236 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:23.624906 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:46:23.624924 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:46:23.626662 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:46:23.628661 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:23.628666 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:23.628683 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:23.629104 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:23.629448 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:46:23.629451 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:46:23.629452 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:46:23.629608 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:46:23.630073 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:46:23.630079 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:46:23.630433 (+ 354us) client_negotiation.cc:769] Sending connection context
0806 02:46:23.630680 (+ 247us) client_negotiation.cc:240] Negotiation successful
0806 02:46:23.631215 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":167,"threads_started":1}
I20260806 02:46:23.633777 17681 heartbeater.cc:347] Connected to a master server at 127.1.183.190:44353
I20260806 02:46:23.634074 17681 heartbeater.cc:464] Registering TS with master...
I20260806 02:46:23.634626 17681 heartbeater.cc:511] Master 127.1.183.190:44353 requested a full tablet report, sending...
I20260806 02:46:23.635912 16980 ts_manager.cc:194] Registered new tserver with Master: 563bce46a5bb469491139329ee43e544 (127.1.183.131:37719)
I20260806 02:46:23.636555 16980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.131@KRBTEST.COM'} at 127.1.183.131:56879
I20260806 02:46:23.644416 1758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:46:23.654304 17038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:23.646502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51896 (local address 127.1.183.190:44353)
0806 02:46:23.646644 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:23.646651 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:23.646824 (+ 173us) server_negotiation.cc:407] Connection header received
0806 02:46:23.646999 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:23.647003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:23.647057 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:23.647145 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:46:23.647874 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.648495 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.649363 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.649589 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.649699 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:23.651099 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:46:23.651103 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:46:23.651106 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:46:23.651137 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:46:23.652480 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:23.652906 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:23.652912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:23.652916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:23.653003 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:46:23.653357 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:46:23.653361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:46:23.653363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:46:23.653611 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:46:23.653760 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:23.654009 (+ 249us) server_negotiation.cc:299] Negotiation successful
0806 02:46:23.654131 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:46:23.657560 16980 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:46:23.661028 16980 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:46:23.676810 17696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:23.670481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42436 (local address 127.1.183.130:38195)
0806 02:46:23.670975 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:23.670989 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:23.671643 (+ 654us) server_negotiation.cc:407] Connection header received
0806 02:46:23.671886 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:23.671894 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:23.672107 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:23.672276 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:46:23.672882 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.673669 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.675332 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.675998 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.676088 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:23.676285 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:23.676521 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 02:46:23.676630 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":210,"threads_started":1}
I20260806 02:46:23.676790 17697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:23.671095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:44147 (local address 127.0.0.1:59672)
0806 02:46:23.671893 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:23.671907 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:23.672031 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:23.672554 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:23.672557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:23.672590 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:46:23.672733 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:23.672740 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.674174 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.674177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:23.675384 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:23.675392 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.676220 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.676224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:23.676365 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:23.676394 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 02:46:23.676488 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 02:46:23.676634 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"mutex_wait_us":130,"thread_start_us":80,"threads_started":1}
I20260806 02:46:23.676808 17695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:23.670302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:38195 (local address 127.0.0.1:42436)
0806 02:46:23.671098 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:23.671540 (+ 442us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:23.671725 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:23.672508 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:23.672513 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:23.672533 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:46:23.672734 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:23.672740 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.673801 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.673804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:23.675199 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:23.675211 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.676132 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.676138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:23.676269 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:23.676352 (+ 83us) client_negotiation.cc:769] Sending connection context
0806 02:46:23.676472 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 02:46:23.676631 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":150,"spinlock_wait_cycles":70272,"thread_start_us":120,"threads_started":1}
I20260806 02:46:23.677335 17699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:23.671235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59672 (local address 127.1.183.129:44147)
0806 02:46:23.671545 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:23.671552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:23.672022 (+ 470us) server_negotiation.cc:407] Connection header received
0806 02:46:23.672187 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:23.672197 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:23.672417 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:23.672832 (+ 415us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:46:23.673002 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.674059 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.675500 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.676036 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.676226 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:23.676383 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:23.677048 (+ 665us) server_negotiation.cc:299] Negotiation successful
0806 02:46:23.677189 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":125,"threads_started":1}
I20260806 02:46:23.678478 17694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:23.670323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:37719 (local address 127.0.0.1:43310)
0806 02:46:23.671508 (+ 1185us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:23.671530 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:23.671653 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:23.672326 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:23.672332 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:23.672357 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:46:23.672525 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:23.672533 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.674043 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.674046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:23.675368 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:23.675379 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.678092 (+ 2713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.678096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:23.678181 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:23.678203 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:46:23.678269 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 02:46:23.678338 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1121,"mutex_wait_us":128,"thread_start_us":93,"threads_started":1}
I20260806 02:46:23.678709 17700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:23.671187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43310 (local address 127.1.183.131:37719)
0806 02:46:23.671734 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:23.671745 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:23.671767 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:46:23.671919 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:23.671923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:23.672107 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:23.672308 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:46:23.672669 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.673886 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.676459 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.677955 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.678094 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:23.678241 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:23.678293 (+ 52us) server_negotiation.cc:299] Negotiation successful
0806 02:46:23.678378 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":163,"threads_started":1}
I20260806 02:46:23.680056 17371 tablet_service.cc:1467] Processing CreateTablet for tablet 6cdd0e23366e4b4fa217fb887d4cd9e0 (DEFAULT_TABLE table=test-table [id=fb639319dbcb42b0acdf3736668e4c09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:46:23.681336 17371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cdd0e23366e4b4fa217fb887d4cd9e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:23.681104 17580 tablet_service.cc:1467] Processing CreateTablet for tablet 6cdd0e23366e4b4fa217fb887d4cd9e0 (DEFAULT_TABLE table=test-table [id=fb639319dbcb42b0acdf3736668e4c09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:46:23.682220 17580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cdd0e23366e4b4fa217fb887d4cd9e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:23.682144 17155 tablet_service.cc:1467] Processing CreateTablet for tablet 6cdd0e23366e4b4fa217fb887d4cd9e0 (DEFAULT_TABLE table=test-table [id=fb639319dbcb42b0acdf3736668e4c09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:46:23.683183 17155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cdd0e23366e4b4fa217fb887d4cd9e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:46:23.688731 17701 tablet_bootstrap.cc:492] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 563bce46a5bb469491139329ee43e544: Bootstrap starting.
I20260806 02:46:23.690064 17702 tablet_bootstrap.cc:492] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 5eed8ed6b0584be3968dbb6263753502: Bootstrap starting.
I20260806 02:46:23.690237 17703 tablet_bootstrap.cc:492] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 724b769ff3624f7c892469118e45120e: Bootstrap starting.
I20260806 02:46:23.692359 17701 tablet_bootstrap.cc:654] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 563bce46a5bb469491139329ee43e544: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:23.692503 17703 tablet_bootstrap.cc:654] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 724b769ff3624f7c892469118e45120e: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:23.692792 17702 tablet_bootstrap.cc:654] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 5eed8ed6b0584be3968dbb6263753502: Neither blocks nor log segments found. Creating new log.
I20260806 02:46:23.693491 17701 log.cc:824] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 563bce46a5bb469491139329ee43e544: Log is configured to *not* fsync() on all Append() calls
I20260806 02:46:23.693491 17703 log.cc:824] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 724b769ff3624f7c892469118e45120e: Log is configured to *not* fsync() on all Append() calls
I20260806 02:46:23.693770 17702 log.cc:824] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 5eed8ed6b0584be3968dbb6263753502: Log is configured to *not* fsync() on all Append() calls
I20260806 02:46:23.695623 17703 tablet_bootstrap.cc:492] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 724b769ff3624f7c892469118e45120e: No bootstrap required, opened a new log
I20260806 02:46:23.695866 17703 ts_tablet_manager.cc:1397] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 724b769ff3624f7c892469118e45120e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:46:23.695959 17702 tablet_bootstrap.cc:492] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 5eed8ed6b0584be3968dbb6263753502: No bootstrap required, opened a new log
I20260806 02:46:23.695951 17701 tablet_bootstrap.cc:492] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 563bce46a5bb469491139329ee43e544: No bootstrap required, opened a new log
I20260806 02:46:23.696180 17702 ts_tablet_manager.cc:1397] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 5eed8ed6b0584be3968dbb6263753502: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.003s
I20260806 02:46:23.696190 17701 ts_tablet_manager.cc:1397] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 563bce46a5bb469491139329ee43e544: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260806 02:46:23.699751 17703 raft_consensus.cc:348] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 724b769ff3624f7c892469118e45120e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eed8ed6b0584be3968dbb6263753502" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 38195 } } peers { permanent_uuid: "563bce46a5bb469491139329ee43e544" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37719 } } peers { permanent_uuid: "724b769ff3624f7c892469118e45120e" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 44147 } }
I20260806 02:46:23.700044 17703 raft_consensus.cc:374] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 724b769ff3624f7c892469118e45120e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:23.700112 17703 raft_consensus.cc:729] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 724b769ff3624f7c892469118e45120e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 724b769ff3624f7c892469118e45120e, State: Initialized, Role: FOLLOWER
I20260806 02:46:23.700271 17701 raft_consensus.cc:348] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 563bce46a5bb469491139329ee43e544 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eed8ed6b0584be3968dbb6263753502" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 38195 } } peers { permanent_uuid: "563bce46a5bb469491139329ee43e544" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37719 } } peers { permanent_uuid: "724b769ff3624f7c892469118e45120e" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 44147 } }
I20260806 02:46:23.700543 17701 raft_consensus.cc:374] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 563bce46a5bb469491139329ee43e544 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:23.700603 17701 raft_consensus.cc:729] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 563bce46a5bb469491139329ee43e544 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 563bce46a5bb469491139329ee43e544, State: Initialized, Role: FOLLOWER
I20260806 02:46:23.700675 17703 consensus_queue.cc:260] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 724b769ff3624f7c892469118e45120e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eed8ed6b0584be3968dbb6263753502" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 38195 } } peers { permanent_uuid: "563bce46a5bb469491139329ee43e544" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37719 } } peers { permanent_uuid: "724b769ff3624f7c892469118e45120e" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 44147 } }
I20260806 02:46:23.701038 17701 consensus_queue.cc:260] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 563bce46a5bb469491139329ee43e544 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eed8ed6b0584be3968dbb6263753502" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 38195 } } peers { permanent_uuid: "563bce46a5bb469491139329ee43e544" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37719 } } peers { permanent_uuid: "724b769ff3624f7c892469118e45120e" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 44147 } }
I20260806 02:46:23.701534 17256 heartbeater.cc:503] Master 127.1.183.190:44353 was elected leader, sending a full tablet report...
I20260806 02:46:23.701392 17702 raft_consensus.cc:348] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 5eed8ed6b0584be3968dbb6263753502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eed8ed6b0584be3968dbb6263753502" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 38195 } } peers { permanent_uuid: "563bce46a5bb469491139329ee43e544" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37719 } } peers { permanent_uuid: "724b769ff3624f7c892469118e45120e" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 44147 } }
I20260806 02:46:23.701858 17702 raft_consensus.cc:374] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 5eed8ed6b0584be3968dbb6263753502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:46:23.701913 17702 raft_consensus.cc:729] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 5eed8ed6b0584be3968dbb6263753502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eed8ed6b0584be3968dbb6263753502, State: Initialized, Role: FOLLOWER
I20260806 02:46:23.701982 17703 ts_tablet_manager.cc:1428] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 724b769ff3624f7c892469118e45120e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 02:46:23.702046 17701 ts_tablet_manager.cc:1428] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 563bce46a5bb469491139329ee43e544: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260806 02:46:23.702307 17702 consensus_queue.cc:260] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 5eed8ed6b0584be3968dbb6263753502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eed8ed6b0584be3968dbb6263753502" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 38195 } } peers { permanent_uuid: "563bce46a5bb469491139329ee43e544" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37719 } } peers { permanent_uuid: "724b769ff3624f7c892469118e45120e" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 44147 } }
I20260806 02:46:23.702479 17681 heartbeater.cc:503] Master 127.1.183.190:44353 was elected leader, sending a full tablet report...
I20260806 02:46:23.702924 17474 heartbeater.cc:503] Master 127.1.183.190:44353 was elected leader, sending a full tablet report...
I20260806 02:46:23.703254 17702 ts_tablet_manager.cc:1428] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 5eed8ed6b0584be3968dbb6263753502: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20260806 02:46:23.752666 17713 raft_consensus.cc:482] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 5eed8ed6b0584be3968dbb6263753502 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:46:23.752918 17713 raft_consensus.cc:504] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 5eed8ed6b0584be3968dbb6263753502 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eed8ed6b0584be3968dbb6263753502" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 38195 } } peers { permanent_uuid: "563bce46a5bb469491139329ee43e544" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37719 } } peers { permanent_uuid: "724b769ff3624f7c892469118e45120e" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 44147 } }
I20260806 02:46:23.754230 17713 leader_election.cc:288] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 5eed8ed6b0584be3968dbb6263753502 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 563bce46a5bb469491139329ee43e544 (127.1.183.131:37719), 724b769ff3624f7c892469118e45120e (127.1.183.129:44147)
I20260806 02:46:23.758731 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:23.754481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:37719 (local address 127.1.183.130:43979)
0806 02:46:23.754608 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:23.754625 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:23.754723 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:23.755031 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:23.755035 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:23.755060 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:46:23.755187 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:23.755194 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.756288 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.756291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:23.757464 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:23.757475 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.758292 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.758295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:23.758384 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:23.758406 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:46:23.758483 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 02:46:23.758576 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260806 02:46:23.758756 17700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:23.754570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:43979 (local address 127.1.183.131:37719)
0806 02:46:23.754695 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:23.754699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:23.754715 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:46:23.754830 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:23.754834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:23.754899 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:23.754989 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:46:23.755345 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.756185 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.757595 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.758171 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.758293 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:23.758352 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:23.758524 (+ 172us) server_negotiation.cc:299] Negotiation successful
0806 02:46:23.758608 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 02:46:23.759133 17699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:23.754631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:43911 (local address 127.1.183.129:44147)
0806 02:46:23.754775 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:23.754778 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:23.755321 (+ 543us) server_negotiation.cc:407] Connection header received
0806 02:46:23.755457 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:23.755460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:23.755511 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:23.755594 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:46:23.755823 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.756496 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.757822 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.758444 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.758596 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:23.758641 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:23.758879 (+ 238us) server_negotiation.cc:299] Negotiation successful
0806 02:46:23.758979 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:46:23.759415 17612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cdd0e23366e4b4fa217fb887d4cd9e0" candidate_uuid: "5eed8ed6b0584be3968dbb6263753502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "563bce46a5bb469491139329ee43e544" is_pre_election: true
I20260806 02:46:23.759509 17714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:23.754495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:44147 (local address 127.1.183.130:43911)
0806 02:46:23.755206 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:23.755221 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:46:23.755350 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:46:23.755607 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:46:23.755609 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:46:23.755622 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:46:23.755714 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:23.755718 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.756601 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.756604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:23.757710 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:46:23.757717 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.758598 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.758601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:46:23.758717 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:23.758742 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 02:46:23.759256 (+ 514us) client_negotiation.cc:240] Negotiation successful
0806 02:46:23.759352 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":313,"thread_start_us":156,"threads_started":1}
I20260806 02:46:23.759790 17612 raft_consensus.cc:2454] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 563bce46a5bb469491139329ee43e544 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5eed8ed6b0584be3968dbb6263753502 in term 0.
I20260806 02:46:23.760281 17180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cdd0e23366e4b4fa217fb887d4cd9e0" candidate_uuid: "5eed8ed6b0584be3968dbb6263753502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "724b769ff3624f7c892469118e45120e" is_pre_election: true
I20260806 02:46:23.760371 17286 leader_election.cc:302] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 5eed8ed6b0584be3968dbb6263753502 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 563bce46a5bb469491139329ee43e544, 5eed8ed6b0584be3968dbb6263753502; no voters:
I20260806 02:46:23.760596 17180 raft_consensus.cc:2454] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 724b769ff3624f7c892469118e45120e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5eed8ed6b0584be3968dbb6263753502 in term 0.
I20260806 02:46:23.760658 17713 raft_consensus.cc:2781] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 5eed8ed6b0584be3968dbb6263753502 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:46:23.760783 17713 raft_consensus.cc:482] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 5eed8ed6b0584be3968dbb6263753502 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:46:23.760840 17713 raft_consensus.cc:3037] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 5eed8ed6b0584be3968dbb6263753502 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:23.762141 17713 raft_consensus.cc:504] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 5eed8ed6b0584be3968dbb6263753502 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eed8ed6b0584be3968dbb6263753502" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 38195 } } peers { permanent_uuid: "563bce46a5bb469491139329ee43e544" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37719 } } peers { permanent_uuid: "724b769ff3624f7c892469118e45120e" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 44147 } }
I20260806 02:46:23.762504 17713 leader_election.cc:288] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 5eed8ed6b0584be3968dbb6263753502 [CANDIDATE]: Term 1 election: Requested vote from peers 563bce46a5bb469491139329ee43e544 (127.1.183.131:37719), 724b769ff3624f7c892469118e45120e (127.1.183.129:44147)
I20260806 02:46:23.762966 17180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cdd0e23366e4b4fa217fb887d4cd9e0" candidate_uuid: "5eed8ed6b0584be3968dbb6263753502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "724b769ff3624f7c892469118e45120e"
I20260806 02:46:23.762997 17612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cdd0e23366e4b4fa217fb887d4cd9e0" candidate_uuid: "5eed8ed6b0584be3968dbb6263753502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "563bce46a5bb469491139329ee43e544"
I20260806 02:46:23.763080 17180 raft_consensus.cc:3037] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 724b769ff3624f7c892469118e45120e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:23.763142 17612 raft_consensus.cc:3037] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 563bce46a5bb469491139329ee43e544 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:46:23.764446 17612 raft_consensus.cc:2454] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 563bce46a5bb469491139329ee43e544 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eed8ed6b0584be3968dbb6263753502 in term 1.
I20260806 02:46:23.764446 17180 raft_consensus.cc:2454] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 724b769ff3624f7c892469118e45120e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eed8ed6b0584be3968dbb6263753502 in term 1.
I20260806 02:46:23.764799 17286 leader_election.cc:302] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 5eed8ed6b0584be3968dbb6263753502 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 563bce46a5bb469491139329ee43e544, 5eed8ed6b0584be3968dbb6263753502; no voters:
I20260806 02:46:23.765030 17713 raft_consensus.cc:2781] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 5eed8ed6b0584be3968dbb6263753502 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:46:23.765260 17713 raft_consensus.cc:686] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 5eed8ed6b0584be3968dbb6263753502 [term 1 LEADER]: Becoming Leader. State: Replica: 5eed8ed6b0584be3968dbb6263753502, State: Running, Role: LEADER
I20260806 02:46:23.765631 17713 consensus_queue.cc:237] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 5eed8ed6b0584be3968dbb6263753502 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eed8ed6b0584be3968dbb6263753502" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 38195 } } peers { permanent_uuid: "563bce46a5bb469491139329ee43e544" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37719 } } peers { permanent_uuid: "724b769ff3624f7c892469118e45120e" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 44147 } }
I20260806 02:46:23.769196 16978 catalog_manager.cc:5495] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 5eed8ed6b0584be3968dbb6263753502 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5eed8ed6b0584be3968dbb6263753502 (127.1.183.130). New cstate: current_term: 1 leader_uuid: "5eed8ed6b0584be3968dbb6263753502" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eed8ed6b0584be3968dbb6263753502" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 38195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "563bce46a5bb469491139329ee43e544" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "724b769ff3624f7c892469118e45120e" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 44147 } health_report { overall_health: UNKNOWN } } }
I20260806 02:46:23.801054 17696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:23.797293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42444 (local address 127.1.183.130:38195)
0806 02:46:23.797458 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 02:46:23.797463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:46:23.797479 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:46:23.797609 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:46:23.797612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:46:23.797677 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:46:23.797790 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:46:23.798072 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.798594 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.799465 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:46:23.799696 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:46:23.799805 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:46:23.800139 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:46:23.800635 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:46:23.800774 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 02:46:23.800815 (+ 41us) server_negotiation.cc:299] Negotiation successful
0806 02:46:23.800872 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 02:46:23.807299 17180 raft_consensus.cc:1261] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 724b769ff3624f7c892469118e45120e [term 1 FOLLOWER]: Refusing update from remote peer 5eed8ed6b0584be3968dbb6263753502: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:46:23.807299 17612 raft_consensus.cc:1261] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 563bce46a5bb469491139329ee43e544 [term 1 FOLLOWER]: Refusing update from remote peer 5eed8ed6b0584be3968dbb6263753502: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:46:23.808073 17713 consensus_queue.cc:1035] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 5eed8ed6b0584be3968dbb6263753502 [LEADER]: Connected to new peer: Peer: permanent_uuid: "563bce46a5bb469491139329ee43e544" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:46:23.808337 17715 consensus_queue.cc:1035] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 5eed8ed6b0584be3968dbb6263753502 [LEADER]: Connected to new peer: Peer: permanent_uuid: "724b769ff3624f7c892469118e45120e" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 44147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:46:23.818608 17720 mvcc.cc:205] Tried to move back new op lower bound from 7315392036066934784 to 7315392035907104768. Current Snapshot: MvccSnapshot[applied={T|T < 7315392036066934784}]
I20260806 02:46:23.825306 16978 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:46:23.825500 16978 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:46:23.827976 16978 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 222237f7a26f401f903c622b1ea9af30: Sending DeleteTablet for 3 replicas of tablet 6cdd0e23366e4b4fa217fb887d4cd9e0
I20260806 02:46:23.828902 17155 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cdd0e23366e4b4fa217fb887d4cd9e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:46:23 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:59672
I20260806 02:46:23.828929 17580 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cdd0e23366e4b4fa217fb887d4cd9e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:46:23 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:43310
I20260806 02:46:23.829558 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 17071
I20260806 02:46:23.829632 17734 tablet_replica.cc:330] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 563bce46a5bb469491139329ee43e544: stopping tablet replica
I20260806 02:46:23.829679 17371 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cdd0e23366e4b4fa217fb887d4cd9e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:46:23 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:42436
I20260806 02:46:23.832681 17734 raft_consensus.cc:2229] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 563bce46a5bb469491139329ee43e544 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:46:23.833104 17734 raft_consensus.cc:2258] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 563bce46a5bb469491139329ee43e544 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:46:23.833607 17735 tablet_replica.cc:330] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 5eed8ed6b0584be3968dbb6263753502: stopping tablet replica
I20260806 02:46:23.834002 17735 raft_consensus.cc:2229] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 5eed8ed6b0584be3968dbb6263753502 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:46:23.834529 17735 raft_consensus.cc:2258] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 5eed8ed6b0584be3968dbb6263753502 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:46:23.837097 17734 ts_tablet_manager.cc:1905] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 563bce46a5bb469491139329ee43e544: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:46:23.840035 17734 ts_tablet_manager.cc:1918] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 563bce46a5bb469491139329ee43e544: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:46:23.840147 17734 log.cc:1196] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 563bce46a5bb469491139329ee43e544: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-2/wal/wals/6cdd0e23366e4b4fa217fb887d4cd9e0
I20260806 02:46:23.840536 17734 ts_tablet_manager.cc:1939] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 563bce46a5bb469491139329ee43e544: Deleting consensus metadata
I20260806 02:46:23.842017 16961 catalog_manager.cc:4841] TS 563bce46a5bb469491139329ee43e544 (127.1.183.131:37719): tablet 6cdd0e23366e4b4fa217fb887d4cd9e0 (table test-table [id=fb639319dbcb42b0acdf3736668e4c09]) successfully deleted
I20260806 02:46:23.845520 17735 ts_tablet_manager.cc:1905] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 5eed8ed6b0584be3968dbb6263753502: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:46:23.848800 17735 ts_tablet_manager.cc:1918] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 5eed8ed6b0584be3968dbb6263753502: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260806 02:46:23.848946 16962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.183.129:44147 (error 108)
I20260806 02:46:23.849009 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 17262
I20260806 02:46:23.849018 17735 log.cc:1196] T 6cdd0e23366e4b4fa217fb887d4cd9e0 P 5eed8ed6b0584be3968dbb6263753502: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984313887056-1758-0/minicluster-data/ts-1/wal/wals/6cdd0e23366e4b4fa217fb887d4cd9e0
I20260806 02:46:23.863034 17694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:23.862642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:44147 (local address 127.0.0.1:59686)
0806 02:46:23.862846 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:23.862931 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.129:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260806 02:46:23.863252 16962 catalog_manager.cc:4584] TS 724b769ff3624f7c892469118e45120e (127.1.183.129:44147): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6cdd0e23366e4b4fa217fb887d4cd9e0: Network error: Client connection negotiation failed: client connection to 127.1.183.129:44147: connect: Connection refused (error 111)
I20260806 02:46:23.869861 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 17481
I20260806 02:46:23.872565 17694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:46:23.870085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:38195 (local address 127.0.0.1:42450)
0806 02:46:23.872418 (+ 2333us) negotiation.cc:107] Waiting for socket to connect
0806 02:46:23.872481 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.130:38195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2228}
I20260806 02:46:23.882743 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 16935
2026-08-06T02:46:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2621 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:46:23 dist-test-slave-gmjp krb5kdc[17741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:46:23 dist-test-slave-gmjp krb5kdc[17741](info): set up 2 sockets
Aug 06 02:46:23 dist-test-slave-gmjp krb5kdc[17741](info): commencing operation
krb5kdc: starting...
W20260806 02:46:25.116983 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.203s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:46:25 dist-test-slave-gmjp krb5kdc[17741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:46:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:46:25Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.183.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.183.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.183.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/rangeradmin_127.1.183.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.183.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/rangeradmin_127.1.183.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.183.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.183.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.183.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/rangerlookup_127.1.183.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.183.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/rangerlookup_127.1.183.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.183.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.183.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.183.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/HTTP_127.1.183.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.183.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/HTTP_127.1.183.148@KRBTEST.COM.keytab.
I20260806 02:46:25.312438 1758 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task0hWVsi/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/postgres -l logfile start
2026-08-06 02:46:27.802 UTC [17782] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 02:46:27.802 UTC [17782] LOG: listening on IPv4 address "127.1.183.148", port 42555
2026-08-06 02:46:27.806 UTC [17782] LOG: listening on Unix socket "/tmp/.s.PGSQL.42555"
2026-08-06 02:46:27.842 UTC [17785] LOG: database system was shut down at 2026-08-06 02:46:26 UTC
2026-08-06 02:46:27.846 UTC [17782] LOG: database system is ready to accept connections
I20260806 02:46:28.979017 1758 mini_postgres.cc:96] Postgres bound to 42555
2026-08-06 02:46:28.987 UTC [17795] FATAL: database "slave" does not exist
127.1.183.148:42555 - accepting connections
I20260806 02:46:28.988703 1758 mini_ranger.cc:161] Starting Ranger...
I20260806 02:46:29.008085 1758 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 02:46:29.511431 1758 mini_ranger.cc:87] Created ranger Postgres database
I20260806 02:46:29.511545 1758 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ranger-admin
2026-08-06 02:46:29,817 [I] DB FLAVOR :POSTGRES
2026-08-06 02:46:29,818 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 02:46:29,818 [I] Checking connection..
2026-08-06 02:46:29,818 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:42555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 02:46:30,102 [I] Checking connection passed.
2026-08-06 02:46:30,102 [I] --------- Verifying version history table ---------
2026-08-06 02:46:30,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:42555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:46:30,371 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 02:46:30,372 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:42555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:46:30,655 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 02:46:30,655 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 02:46:30,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:42555/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 02:46:30.928 UTC [17882] WARNING: there is no transaction in progress
2026-08-06 02:46:30,936 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 02:46:30,936 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:42555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:46:31,189 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 02:46:31,189 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 02:46:31,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:42555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 02:46:31,470 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:42555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 02:46:31,743 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:42555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gmjp.c.gcp-upstream.internal','N') ;"
2026-08-06 02:46:31,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:42555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 02:46:32,253 [I] Table x_portal_user does not exist in database ranger
2026-08-06 02:46:32,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:42555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 02:46:32,534 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 02:46:32,534 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:42555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 02:46:32,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:42555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 02:46:33,060 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 02:46:33,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:42555/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 02:46:34.083 UTC [18038] WARNING: there is no transaction in progress
2026-08-06 02:46:34.101 UTC [18038] WARNING: there is no transaction in progress
2026-08-06 02:46:34.117 UTC [18038] WARNING: there is no transaction in progress
2026-08-06 02:46:34.133 UTC [18038] WARNING: there is no transaction in progress
2026-08-06 02:46:34.151 UTC [18038] WARNING: there is no transaction in progress
2026-08-06 02:46:34.170 UTC [18038] WARNING: there is no transaction in progress
2026-08-06 02:46:34.226 UTC [18038] WARNING: there is no transaction in progress
2026-08-06 02:46:34.233 UTC [18038] WARNING: there is no transaction in progress
2026-08-06 02:46:34.249 UTC [18038] WARNING: there is no transaction in progress
2026-08-06 02:46:34.260 UTC [18038] WARNING: there is no transaction in progress
2026-08-06 02:46:34.271 UTC [18038] WARNING: there is no transaction in progress
2026-08-06 02:46:34.285 UTC [18038] WARNING: there is no transaction in progress
2026-08-06 02:46:34.296 UTC [18038] WARNING: there is no transaction in progress
2026-08-06 02:46:34.302 UTC [18038] WARNING: there is no transaction in progress
2026-08-06 02:46:34.310 UTC [18038] WARNING: there is no transaction in progress
2026-08-06 02:46:34.318 UTC [18038] WARNING: there is no transaction in progress
2026-08-06 02:46:35,003 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 02:46:35,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:42555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 02:46:35,276 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 02:46:35,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:42555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 02:46:35,558 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 02:46:35,558 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:42555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 02:46:35,832 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 02:46:35,833 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:42555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 02:46:36,112 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:42555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 02:46:36,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:42555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gmjp.c.gcp-upstream.internal';"
2026-08-06 02:46:36,651 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 02:46:36,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:42555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 02:46:36,927 [I] DB_PATCHES have already been applied
I20260806 02:46:36.933791 1758 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ranger-admin:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task0hWVsi/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 02:46:36.933863 1758 mini_ranger.cc:193] Using host: 127.1.183.148
I20260806 02:46:36.938578 1758 mini_ranger.cc:227] Ranger admin URL: http://127.1.183.148:38285
Aug 06, 2026 2:46:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 2:46:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 2:46:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 2:46:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 2:46:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 2:46:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 2:46:38 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 02:46:38.331251 1758 mini_ranger.cc:160] Time spent starting Ranger: real 9.343s user 0.000s sys 0.014s
Aug 06, 2026 2:46:59 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 2:46:59 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 06, 2026 2:46:59 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 2:46:59 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 2:46:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 2:46:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 2:46:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 2:46:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 2:46:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 2:46:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 2:46:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 2:46:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 2:46:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 2:46:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 2:46:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 2:46:59 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 2:46:59 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 2:47:00 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 02:47:01.436641 1758 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.183.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.183.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.183.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/rangerkms_127.1.183.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.183.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/rangerkms_127.1.183.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.183.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.183.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.183.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/HTTP_127.1.183.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.183.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/HTTP_127.1.183.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 02:47:01 dist-test-slave-gmjp krb5kdc[17741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984421, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 02:47:01.531141 1758 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 02:47:01.546775 1758 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 02:47:02.428706 1758 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 02:47:02.428819 1758 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ranger-kms
I20260806 02:47:02.428953 1758 mini_ranger_kms.cc:227] Using postgres at 127.1.183.148:42555
2026-08-06 02:47:02,698 [I] DB FLAVOR :POSTGRES
2026-08-06 02:47:02,698 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 02:47:02,698 [I] Checking connection
2026-08-06 02:47:02,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:42555/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 02:47:02,952 [I] connection success
2026-08-06 02:47:02,952 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 02:47:02,952 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 02:47:02,952 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:42555/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 02:47:03,204 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 02:47:03,204 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 02:47:03,204 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 02:47:03,204 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:42555/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 02:47:03,504 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 02:47:04.071707 1758 mini_ranger_kms.cc:310] Created kms service
I20260806 02:47:04.277002 1758 mini_ranger_kms.cc:326] Created kudu user
I20260806 02:47:04.330147 1758 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 02:47:04 dist-test-slave-gmjp krb5kdc[17741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984421, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.183.148@KRBTEST.COM
I20260806 02:47:04.782845 1758 mini_ranger_kms.cc:384] Added ranger policy
I20260806 02:47:04.783016 1758 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ranger-kms:/tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 02:47:04.783038 1758 mini_ranger_kms.cc:240] Using host: 127.1.183.148
I20260806 02:47:04.789407 1758 mini_ranger_kms.cc:276] Ranger KMS PID: 18329
I20260806 02:47:04.789600 1758 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.183.148:37905
Aug 06, 2026 2:47:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 2:47:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 2:47:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 2:47:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 2:47:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 2:47:06 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 02:47:07 dist-test-slave-gmjp krb5kdc[17741](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785984427, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.183.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:47:07.526628 1758 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.996s user 0.001s sys 0.252s
I20260806 02:47:07.526774 1758 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 02:47:07.526892 1758 mini_ranger_kms.cc:401] 127.1.183.148:37905/kms/v1/keys
Aug 06 02:47:07 dist-test-slave-gmjp krb5kdc[17741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984427, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.183.148@KRBTEST.COM for HTTP/127.1.183.148@KRBTEST.COM
Aug 06 02:47:08 dist-test-slave-gmjp krb5kdc[17741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984427, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.183.148@KRBTEST.COM for HTTP/127.1.183.148@KRBTEST.COM
Aug 06 02:47:10 dist-test-slave-gmjp krb5kdc[17741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
I20260806 02:47:10.493013 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:33895
--webserver_interface=127.1.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:38835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:33895
--ranger_config_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.183.148:37905/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:47:10.616194 18402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:47:10.616562 18402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:47:10.616631 18402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:47:10.621196 18402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:47:10.621286 18402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:47:10.621306 18402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:47:10.621323 18402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:47:10.621338 18402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:47:10.627326 18402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:38835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.183.148:37905/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:33895
--ranger_config_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:33895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:47:10.628618 18402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:47:10.629719 18402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:47:10.639485 18402 server_base.cc:1034] running on GCE node
W20260806 02:47:10.639508 18408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:47:10.639782 18414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:47:10.639798 18411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:47:10.640631 18402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:47:10.641878 18402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:47:10.643066 18402 hybrid_clock.cc:648] HybridClock initialized: now 1785984430643042 us; error 47 us; skew 500 ppm
Aug 06 02:47:10 dist-test-slave-gmjp krb5kdc[17741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984430, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:47:10.651095 18402 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:47:10.652309 18402 webserver.cc:460] Webserver started at http://127.1.183.190:39027/ using document root <none> and password file <none>
I20260806 02:47:10.652944 18402 fs_manager.cc:359] Metadata directory not provided
I20260806 02:47:10.653009 18402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:47:10.653272 18402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:47:10 dist-test-slave-gmjp krb5kdc[17741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984430, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for HTTP/127.1.183.148@KRBTEST.COM
I20260806 02:47:10.714993 18402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "ba889e2cbfd74dcab01bc48a39d72338"
format_stamp: "Formatted at 2026-08-06 02:47:10 on dist-test-slave-gmjp"
server_key: "547ae7b7c7083e7a5451479e0b85ba36"
server_key_iv: "2653e634d57d525653e6d1d35380cdf7"
server_key_version: "kuduclusterkey@0"
I20260806 02:47:10.715834 18402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "ba889e2cbfd74dcab01bc48a39d72338"
format_stamp: "Formatted at 2026-08-06 02:47:10 on dist-test-slave-gmjp"
server_key: "547ae7b7c7083e7a5451479e0b85ba36"
server_key_iv: "2653e634d57d525653e6d1d35380cdf7"
server_key_version: "kuduclusterkey@0"
I20260806 02:47:10.721505 18402 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 02:47:10.738137 18421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:47:10.740043 18402 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260806 02:47:10.740259 18402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "ba889e2cbfd74dcab01bc48a39d72338"
format_stamp: "Formatted at 2026-08-06 02:47:10 on dist-test-slave-gmjp"
server_key: "547ae7b7c7083e7a5451479e0b85ba36"
server_key_iv: "2653e634d57d525653e6d1d35380cdf7"
server_key_version: "kuduclusterkey@0"
I20260806 02:47:10.740428 18402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:47:10.757820 18402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:47:10.761157 18402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:47:10.761377 18402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:47:10.770221 18402 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:33895
I20260806 02:47:10.770288 18511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:33895 every 8 connection(s)
I20260806 02:47:10.771344 18402 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:47:10.772666 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 18402
I20260806 02:47:10.772804 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:47:10.773147 1758 external_mini_cluster.cc:1428] Setting key 7e50cd9ded2214507e7b6db421af901c
I20260806 02:47:10.775986 18512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:47:10.783390 18512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba889e2cbfd74dcab01bc48a39d72338: Bootstrap starting.
Aug 06 02:47:10 dist-test-slave-gmjp krb5kdc[17741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:47:10.786417 18512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba889e2cbfd74dcab01bc48a39d72338: Neither blocks nor log segments found. Creating new log.
I20260806 02:47:10.787454 18512 log.cc:824] T 00000000000000000000000000000000 P ba889e2cbfd74dcab01bc48a39d72338: Log is configured to *not* fsync() on all Append() calls
I20260806 02:47:10.790262 18512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba889e2cbfd74dcab01bc48a39d72338: No bootstrap required, opened a new log
I20260806 02:47:10.793481 18515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:10.774692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60510 (local address 127.1.183.190:33895)
0806 02:47:10.775159 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0806 02:47:10.775184 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 02:47:10.775233 (+ 49us) server_negotiation.cc:407] Connection header received
0806 02:47:10.776139 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:47:10.776206 (+ 67us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:47:10.776677 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:47:10.777078 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:47:10.779856 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:10.781643 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:10.782887 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:10.783590 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:10.783759 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:10.787058 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:47:10.787066 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:47:10.787085 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:47:10.787124 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:47:10.789963 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:47:10.790547 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:47:10.790554 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:47:10.790562 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:47:10.790715 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:47:10.791041 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:47:10.791044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:47:10.791046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:47:10.791585 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:47:10.791786 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0806 02:47:10.792108 (+ 322us) server_negotiation.cc:299] Negotiation successful
0806 02:47:10.792459 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":133,"threads_started":1}
I20260806 02:47:10.794678 18512 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba889e2cbfd74dcab01bc48a39d72338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba889e2cbfd74dcab01bc48a39d72338" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33895 } }
I20260806 02:47:10.794968 18512 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba889e2cbfd74dcab01bc48a39d72338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:47:10.795073 18512 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba889e2cbfd74dcab01bc48a39d72338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba889e2cbfd74dcab01bc48a39d72338, State: Initialized, Role: FOLLOWER
I20260806 02:47:10.795660 18512 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba889e2cbfd74dcab01bc48a39d72338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba889e2cbfd74dcab01bc48a39d72338" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33895 } }
I20260806 02:47:10.795826 18512 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba889e2cbfd74dcab01bc48a39d72338 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:47:10.795917 18512 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba889e2cbfd74dcab01bc48a39d72338 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:47:10.796023 18512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba889e2cbfd74dcab01bc48a39d72338 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:47:10.797108 18512 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba889e2cbfd74dcab01bc48a39d72338 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba889e2cbfd74dcab01bc48a39d72338" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33895 } }
I20260806 02:47:10.797505 18512 leader_election.cc:302] T 00000000000000000000000000000000 P ba889e2cbfd74dcab01bc48a39d72338 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba889e2cbfd74dcab01bc48a39d72338; no voters:
I20260806 02:47:10.797883 18512 leader_election.cc:288] T 00000000000000000000000000000000 P ba889e2cbfd74dcab01bc48a39d72338 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:47:10.797987 18517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba889e2cbfd74dcab01bc48a39d72338 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:47:10.798269 18517 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba889e2cbfd74dcab01bc48a39d72338 [term 1 LEADER]: Becoming Leader. State: Replica: ba889e2cbfd74dcab01bc48a39d72338, State: Running, Role: LEADER
I20260806 02:47:10.798741 18517 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba889e2cbfd74dcab01bc48a39d72338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba889e2cbfd74dcab01bc48a39d72338" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33895 } }
I20260806 02:47:10.799378 18512 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba889e2cbfd74dcab01bc48a39d72338 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:47:10.800064 18517 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba889e2cbfd74dcab01bc48a39d72338 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba889e2cbfd74dcab01bc48a39d72338. Latest consensus state: current_term: 1 leader_uuid: "ba889e2cbfd74dcab01bc48a39d72338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba889e2cbfd74dcab01bc48a39d72338" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33895 } } }
I20260806 02:47:10.800313 18517 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba889e2cbfd74dcab01bc48a39d72338 [sys.catalog]: This master's current role is: LEADER
I20260806 02:47:10.800604 18518 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba889e2cbfd74dcab01bc48a39d72338 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba889e2cbfd74dcab01bc48a39d72338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba889e2cbfd74dcab01bc48a39d72338" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 33895 } } }
I20260806 02:47:10.800712 18518 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba889e2cbfd74dcab01bc48a39d72338 [sys.catalog]: This master's current role is: LEADER
I20260806 02:47:10.802031 18512 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 02:47:10.806821 18528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:47:10.812615 18528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:47:10.822314 18528 catalog_manager.cc:1347] Generated new cluster ID: 4c084e0d52e34f4e900d659dbdb32b60
I20260806 02:47:10.822398 18528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:47:10.848347 18528 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:47:10.849923 18528 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:47:10.858495 18528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba889e2cbfd74dcab01bc48a39d72338: Generated new TSK 0
I20260806 02:47:10.859680 18528 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
I20260806 02:47:11.660099 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.183.148:37905/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.129:0
--local_ip_for_outbound_sockets=127.1.183.129
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:33895
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:38835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:47:11.827296 18578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:47:11.827665 18578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:47:11.827801 18578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:47:11.832686 18578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:47:11.832784 18578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:47:11.832876 18578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.129
I20260806 02:47:11.841593 18578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:38835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.183.148:37905/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:33895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.129
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:47:11.843060 18578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:47:11.844502 18578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:47:11.856639 18585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:47:11.856633 18586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:47:11.857357 18589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:47:11.857734 18578 server_base.cc:1034] running on GCE node
I20260806 02:47:11.858258 18578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:47:11.859206 18578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:47:11.860481 18578 hybrid_clock.cc:648] HybridClock initialized: now 1785984431860446 us; error 50 us; skew 500 ppm
Aug 06 02:47:11 dist-test-slave-gmjp krb5kdc[17741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984431, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:47:11.864579 18578 init.cc:377] Logged in from keytab as kudu/127.1.183.129@KRBTEST.COM (short username kudu)
I20260806 02:47:11.865890 18578 webserver.cc:460] Webserver started at http://127.1.183.129:39699/ using document root <none> and password file <none>
I20260806 02:47:11.866747 18578 fs_manager.cc:359] Metadata directory not provided
I20260806 02:47:11.866932 18578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:47:11.867274 18578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:47:11 dist-test-slave-gmjp krb5kdc[17741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984431, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for HTTP/127.1.183.148@KRBTEST.COM
I20260806 02:47:11.891664 18578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-0/data/instance:
uuid: "aee1ab25738c45c3a85a8497dcca5547"
format_stamp: "Formatted at 2026-08-06 02:47:11 on dist-test-slave-gmjp"
server_key: "f3fec875194a665d07285752a26eff85"
server_key_iv: "f2e756e5d445a40e97da17be9e077bec"
server_key_version: "kuduclusterkey@0"
I20260806 02:47:11.892633 18578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance:
uuid: "aee1ab25738c45c3a85a8497dcca5547"
format_stamp: "Formatted at 2026-08-06 02:47:11 on dist-test-slave-gmjp"
server_key: "f3fec875194a665d07285752a26eff85"
server_key_iv: "f2e756e5d445a40e97da17be9e077bec"
server_key_version: "kuduclusterkey@0"
I20260806 02:47:11.898542 18578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 02:47:11.918150 18603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:47:11.920107 18578 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260806 02:47:11.920346 18578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-0/wal
uuid: "aee1ab25738c45c3a85a8497dcca5547"
format_stamp: "Formatted at 2026-08-06 02:47:11 on dist-test-slave-gmjp"
server_key: "f3fec875194a665d07285752a26eff85"
server_key_iv: "f2e756e5d445a40e97da17be9e077bec"
server_key_version: "kuduclusterkey@0"
I20260806 02:47:11.920568 18578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:47:11.939816 18578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:47:11.944088 18578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:47:11.944458 18578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:47:11.945271 18578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:47:11.946377 18578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:47:11.946463 18578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:47:11.946560 18578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:47:11.946622 18578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:47:11.958442 18578 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.129:44451
I20260806 02:47:11.958655 18771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.129:44451 every 8 connection(s)
I20260806 02:47:11.959661 18578 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
I20260806 02:47:11.969139 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 18578
I20260806 02:47:11.969295 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance
I20260806 02:47:11.969645 1758 external_mini_cluster.cc:1428] Setting key d9d4e25f33604c772d027d788844d5af
Aug 06 02:47:11 dist-test-slave-gmjp krb5kdc[17741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984431, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:47:11.975107 18780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:11.961766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:40081 (local address 127.1.183.190:33895)
0806 02:47:11.962103 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0806 02:47:11.962108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:47:11.962861 (+ 753us) server_negotiation.cc:407] Connection header received
0806 02:47:11.963883 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:47:11.963886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:47:11.963946 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:47:11.964043 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:47:11.965498 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:11.966063 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:11.967229 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:11.967477 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:11.967595 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:11.970455 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:47:11.970459 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:47:11.970461 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:47:11.970491 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:47:11.972487 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:47:11.973196 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:47:11.973203 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:47:11.973208 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:47:11.973306 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:47:11.973768 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:47:11.973774 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:47:11.973779 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:47:11.974048 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:47:11.974131 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 02:47:11.974773 (+ 642us) server_negotiation.cc:299] Negotiation successful
0806 02:47:11.974915 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":109,"threads_started":1}
I20260806 02:47:11.976125 18781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:11.962215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:33895 (local address 127.1.183.129:40081)
0806 02:47:11.962701 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0806 02:47:11.962754 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:47:11.963649 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:47:11.964383 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:47:11.964390 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:47:11.964678 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:47:11.965282 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:47:11.965302 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:47:11.966206 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:47:11.966210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:47:11.967101 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:47:11.967110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:47:11.967643 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:47:11.967648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:47:11.967827 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:11.968553 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:47:11.968571 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:47:11.970273 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:47:11.972633 (+ 2360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:47:11.972644 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:47:11.972665 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:47:11.973069 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:47:11.973441 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:47:11.973446 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:47:11.973451 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:47:11.973654 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:47:11.974203 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:47:11.974209 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:47:11.974516 (+ 307us) client_negotiation.cc:769] Sending connection context
0806 02:47:11.974732 (+ 216us) client_negotiation.cc:240] Negotiation successful
0806 02:47:11.975080 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":134,"threads_started":1}
I20260806 02:47:11.978050 18775 heartbeater.cc:347] Connected to a master server at 127.1.183.190:33895
I20260806 02:47:11.978307 18775 heartbeater.cc:464] Registering TS with master...
I20260806 02:47:11.978861 18775 heartbeater.cc:511] Master 127.1.183.190:33895 requested a full tablet report, sending...
I20260806 02:47:11.980650 18452 ts_manager.cc:194] Registered new tserver with Master: aee1ab25738c45c3a85a8497dcca5547 (127.1.183.129:44451)
I20260806 02:47:11.982312 18452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.129@KRBTEST.COM'} at 127.1.183.129:40081
WARNING: no policy specified for kudu/127.1.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
I20260806 02:47:12.011636 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.183.148:37905/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.130:0
--local_ip_for_outbound_sockets=127.1.183.130
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:33895
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:38835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:47:12 dist-test-slave-gmjp krb5kdc[17741](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785984432, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 02:47:12.173775 18786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:47:12.174180 18786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:47:12.174247 18786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:47:12.180673 18786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:47:12.180773 18786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:47:12.180866 18786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.130
I20260806 02:47:12.189332 18786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:38835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.183.148:37905/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:33895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.130
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:47:12.190913 18786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:47:12.192456 18786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:47:12.206111 18799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:47:12.206111 18798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:47:12.206416 18803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:47:12.207712 18786 server_base.cc:1034] running on GCE node
I20260806 02:47:12.208284 18786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:47:12.209153 18786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:47:12.210366 18786 hybrid_clock.cc:648] HybridClock initialized: now 1785984432210344 us; error 39 us; skew 500 ppm
Aug 06 02:47:12 dist-test-slave-gmjp krb5kdc[17741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984432, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:47:12.214268 18786 init.cc:377] Logged in from keytab as kudu/127.1.183.130@KRBTEST.COM (short username kudu)
I20260806 02:47:12.215746 18786 webserver.cc:460] Webserver started at http://127.1.183.130:39057/ using document root <none> and password file <none>
I20260806 02:47:12.216658 18786 fs_manager.cc:359] Metadata directory not provided
I20260806 02:47:12.216768 18786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:47:12.217124 18786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:47:12 dist-test-slave-gmjp krb5kdc[17741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984432, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for HTTP/127.1.183.148@KRBTEST.COM
I20260806 02:47:12.235682 18786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-1/data/instance:
uuid: "e628a841cc774eaa859cc32345bb5fa2"
format_stamp: "Formatted at 2026-08-06 02:47:12 on dist-test-slave-gmjp"
server_key: "1c2c93c17bf458fe2326b278f3d9be80"
server_key_iv: "965da0ebc2b2070b674fd11966798d8c"
server_key_version: "kuduclusterkey@0"
I20260806 02:47:12.236935 18786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance:
uuid: "e628a841cc774eaa859cc32345bb5fa2"
format_stamp: "Formatted at 2026-08-06 02:47:12 on dist-test-slave-gmjp"
server_key: "1c2c93c17bf458fe2326b278f3d9be80"
server_key_iv: "965da0ebc2b2070b674fd11966798d8c"
server_key_version: "kuduclusterkey@0"
I20260806 02:47:12.254096 18786 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.004s sys 0.000s
I20260806 02:47:12.271809 18815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:47:12.273557 18786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:47:12.273743 18786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-1/wal
uuid: "e628a841cc774eaa859cc32345bb5fa2"
format_stamp: "Formatted at 2026-08-06 02:47:12 on dist-test-slave-gmjp"
server_key: "1c2c93c17bf458fe2326b278f3d9be80"
server_key_iv: "965da0ebc2b2070b674fd11966798d8c"
server_key_version: "kuduclusterkey@0"
I20260806 02:47:12.273905 18786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:47:12.307919 18786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:47:12.311477 18786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:47:12.311710 18786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:47:12.312544 18786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:47:12.313655 18786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:47:12.313750 18786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:47:12.313825 18786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:47:12.313882 18786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:47:12.326321 18786 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.130:38763
I20260806 02:47:12.327600 18786 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
I20260806 02:47:12.329193 18974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.130:38763 every 8 connection(s)
I20260806 02:47:12.334657 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 18786
I20260806 02:47:12.334820 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance
I20260806 02:47:12.335206 1758 external_mini_cluster.cc:1428] Setting key 3606b9eb51de72d4090c9852d9f394aa
Aug 06 02:47:12 dist-test-slave-gmjp krb5kdc[17741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984432, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:47:12.349684 18780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:12.331293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:48051 (local address 127.1.183.190:33895)
0806 02:47:12.331460 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 02:47:12.331464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:47:12.332665 (+ 1201us) server_negotiation.cc:407] Connection header received
0806 02:47:12.334023 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:47:12.334027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:47:12.334092 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:47:12.334153 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:47:12.337641 (+ 3488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:12.338504 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:12.340436 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:12.340788 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:12.340878 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:12.344516 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:47:12.344521 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:47:12.344524 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:47:12.344558 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:47:12.346657 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:47:12.347450 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:47:12.347454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:47:12.347456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:47:12.347565 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:47:12.348054 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:47:12.348060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:47:12.348062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:47:12.348365 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:47:12.348476 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 02:47:12.349296 (+ 820us) server_negotiation.cc:299] Negotiation successful
0806 02:47:12.349476 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:47:12.351867 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:12.331768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:33895 (local address 127.1.183.130:48051)
0806 02:47:12.332466 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0806 02:47:12.332535 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0806 02:47:12.333720 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:47:12.334629 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:47:12.334640 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:47:12.335182 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:47:12.337439 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:47:12.337469 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:47:12.338694 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:47:12.338700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:47:12.340257 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:47:12.340271 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:47:12.341135 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:47:12.341141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:47:12.341375 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:12.342187 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:47:12.342211 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:47:12.344236 (+ 2025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:47:12.346823 (+ 2587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:47:12.346832 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:47:12.346855 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:47:12.347293 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:47:12.347699 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:47:12.347704 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:47:12.347706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:47:12.347925 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:47:12.348556 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:47:12.348564 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:47:12.348954 (+ 390us) client_negotiation.cc:769] Sending connection context
0806 02:47:12.350245 (+ 1291us) client_negotiation.cc:240] Negotiation successful
0806 02:47:12.350679 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":173,"threads_started":1}
I20260806 02:47:12.353699 18975 heartbeater.cc:347] Connected to a master server at 127.1.183.190:33895
I20260806 02:47:12.353976 18975 heartbeater.cc:464] Registering TS with master...
I20260806 02:47:12.354636 18975 heartbeater.cc:511] Master 127.1.183.190:33895 requested a full tablet report, sending...
I20260806 02:47:12.355939 18452 ts_manager.cc:194] Registered new tserver with Master: e628a841cc774eaa859cc32345bb5fa2 (127.1.183.130:38763)
I20260806 02:47:12.356762 18452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.130@KRBTEST.COM'} at 127.1.183.130:48051
WARNING: no policy specified for kudu/127.1.183.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
I20260806 02:47:12.399426 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.183.148:37905/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.131:0
--local_ip_for_outbound_sockets=127.1.183.131
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:33895
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:38835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:47:12 dist-test-slave-gmjp krb5kdc[17741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984432, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for HTTP/127.1.183.148@KRBTEST.COM
W20260806 02:47:12.581569 18982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:47:12.581988 18982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:47:12.582057 18982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:47:12.588743 18982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:47:12.588831 18982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:47:12.588922 18982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.131
I20260806 02:47:12.597365 18982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:38835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.183.148:37905/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:33895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.131
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:47:12.598968 18982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:47:12.600731 18982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:47:12.615288 18989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:47:12.615302 18988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:47:12.615316 18991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:47:12.616420 18982 server_base.cc:1034] running on GCE node
I20260806 02:47:12.617075 18982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:47:12.617954 18982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:47:12.636358 18982 hybrid_clock.cc:648] HybridClock initialized: now 1785984432636226 us; error 152 us; skew 500 ppm
Aug 06 02:47:12 dist-test-slave-gmjp krb5kdc[17741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984432, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:47:12.640625 18982 init.cc:377] Logged in from keytab as kudu/127.1.183.131@KRBTEST.COM (short username kudu)
I20260806 02:47:12.642010 18982 webserver.cc:460] Webserver started at http://127.1.183.131:42895/ using document root <none> and password file <none>
I20260806 02:47:12.642892 18982 fs_manager.cc:359] Metadata directory not provided
I20260806 02:47:12.643009 18982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:47:12.643314 18982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:47:12 dist-test-slave-gmjp krb5kdc[17741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984432, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for HTTP/127.1.183.148@KRBTEST.COM
I20260806 02:47:12.662106 18982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-2/data/instance:
uuid: "e5afcb2024bf422492ccd28cbcf470c9"
format_stamp: "Formatted at 2026-08-06 02:47:12 on dist-test-slave-gmjp"
server_key: "18ad8542ad7bb2eff941271126fc5105"
server_key_iv: "3b8a9b803b72bee4db1f2a281e916eac"
server_key_version: "kuduclusterkey@0"
I20260806 02:47:12.663121 18982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance:
uuid: "e5afcb2024bf422492ccd28cbcf470c9"
format_stamp: "Formatted at 2026-08-06 02:47:12 on dist-test-slave-gmjp"
server_key: "18ad8542ad7bb2eff941271126fc5105"
server_key_iv: "3b8a9b803b72bee4db1f2a281e916eac"
server_key_version: "kuduclusterkey@0"
I20260806 02:47:12.674019 18982 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.004s sys 0.000s
I20260806 02:47:12.712675 18999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:47:12.714656 18982 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.004s sys 0.000s
I20260806 02:47:12.714829 18982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-2/wal
uuid: "e5afcb2024bf422492ccd28cbcf470c9"
format_stamp: "Formatted at 2026-08-06 02:47:12 on dist-test-slave-gmjp"
server_key: "18ad8542ad7bb2eff941271126fc5105"
server_key_iv: "3b8a9b803b72bee4db1f2a281e916eac"
server_key_version: "kuduclusterkey@0"
I20260806 02:47:12.714983 18982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:47:12.734058 18982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:47:12.737818 18982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:47:12.738086 18982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:47:12.738829 18982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:47:12.739953 18982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:47:12.740075 18982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:47:12.740224 18982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:47:12.740291 18982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:47:12.757673 18982 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.131:45109
I20260806 02:47:12.759022 18982 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
I20260806 02:47:12.768594 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 18982
I20260806 02:47:12.768746 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance
I20260806 02:47:12.769060 1758 external_mini_cluster.cc:1428] Setting key 3287af68875198c5d36b0d3b0cd67b2f
I20260806 02:47:12.760723 19175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.131:45109 every 8 connection(s)
Aug 06 02:47:12 dist-test-slave-gmjp krb5kdc[17741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984432, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:47:12.793395 18780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:12.764392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:43553 (local address 127.1.183.190:33895)
0806 02:47:12.764523 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 02:47:12.764528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:47:12.765968 (+ 1440us) server_negotiation.cc:407] Connection header received
0806 02:47:12.767854 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:47:12.767858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:47:12.767918 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:47:12.767989 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:47:12.777505 (+ 9516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:12.778303 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:12.780309 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:12.780648 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:12.780731 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:12.788077 (+ 7346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:47:12.788082 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:47:12.788085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:47:12.788124 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:47:12.790165 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:47:12.790973 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:47:12.790977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:47:12.790979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:47:12.791075 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:47:12.791630 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:47:12.791634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:47:12.791635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:47:12.792031 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:47:12.792183 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 02:47:12.793077 (+ 894us) server_negotiation.cc:299] Negotiation successful
0806 02:47:12.793219 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 02:47:12.796190 19180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:12.763557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:33895 (local address 127.1.183.131:43553)
0806 02:47:12.765785 (+ 2228us) negotiation.cc:107] Waiting for socket to connect
0806 02:47:12.765843 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 02:47:12.767568 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:47:12.768457 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:47:12.768467 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:47:12.769020 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:47:12.769943 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:47:12.769963 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:47:12.778568 (+ 8605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:47:12.778572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:47:12.780125 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:47:12.780141 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:47:12.780983 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:47:12.780987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:47:12.781217 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:12.782220 (+ 1003us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:47:12.782247 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:47:12.787758 (+ 5511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:47:12.790370 (+ 2612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:47:12.790379 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:47:12.790404 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:47:12.790832 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:47:12.791189 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:47:12.791192 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:47:12.791194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:47:12.791465 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:47:12.792341 (+ 876us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:47:12.792349 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:47:12.792749 (+ 400us) client_negotiation.cc:769] Sending connection context
0806 02:47:12.793917 (+ 1168us) client_negotiation.cc:240] Negotiation successful
0806 02:47:12.794651 (+ 734us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1975,"thread_start_us":269,"threads_started":1}
I20260806 02:47:12.797832 19176 heartbeater.cc:347] Connected to a master server at 127.1.183.190:33895
I20260806 02:47:12.798100 19176 heartbeater.cc:464] Registering TS with master...
I20260806 02:47:12.798731 19176 heartbeater.cc:511] Master 127.1.183.190:33895 requested a full tablet report, sending...
I20260806 02:47:12.799973 18452 ts_manager.cc:194] Registered new tserver with Master: e5afcb2024bf422492ccd28cbcf470c9 (127.1.183.131:45109)
I20260806 02:47:12.800720 18452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.131@KRBTEST.COM'} at 127.1.183.131:43553
I20260806 02:47:12.807821 1758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:47:12.822726 18780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:12.811164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60184 (local address 127.1.183.190:33895)
0806 02:47:12.811287 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 02:47:12.811292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:47:12.811966 (+ 674us) server_negotiation.cc:407] Connection header received
0806 02:47:12.812204 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:47:12.812208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:47:12.812321 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:47:12.812376 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:47:12.813981 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:12.814763 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:12.815971 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:12.816314 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:12.816392 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:12.818301 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:47:12.818306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:47:12.818309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:47:12.818340 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:47:12.820201 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:47:12.820813 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:47:12.820817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:47:12.820818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:47:12.820938 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:47:12.821268 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:47:12.821271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:47:12.821273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:47:12.821521 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:47:12.821643 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 02:47:12.822411 (+ 768us) server_negotiation.cc:299] Negotiation successful
0806 02:47:12.822554 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 02:47:12.827011 18452 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:47:12.831642 18452 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:47:12.860553 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:12.849570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:44451 (local address 127.0.0.1:50312)
0806 02:47:12.852574 (+ 3004us) negotiation.cc:107] Waiting for socket to connect
0806 02:47:12.852585 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 02:47:12.852657 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:47:12.853654 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:47:12.853658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:47:12.854444 (+ 786us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:47:12.854554 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:47:12.854560 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:47:12.856068 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:47:12.856071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:47:12.857490 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:47:12.857503 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:47:12.859921 (+ 2418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:47:12.859939 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:47:12.860077 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:12.860121 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 02:47:12.860257 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 02:47:12.860386 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2956,"thread_start_us":198,"threads_started":1}
I20260806 02:47:12.863433 19212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:12.850091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50312 (local address 127.1.183.129:44451)
0806 02:47:12.852183 (+ 2092us) server_negotiation.cc:206] Beginning negotiation
0806 02:47:12.852190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:47:12.852747 (+ 557us) server_negotiation.cc:407] Connection header received
0806 02:47:12.852839 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:47:12.852844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:47:12.853097 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:47:12.853248 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:47:12.854690 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:12.855829 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:12.858720 (+ 2891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:12.859596 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:12.859700 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:12.859855 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 02:47:12.863140 (+ 3285us) server_negotiation.cc:299] Negotiation successful
0806 02:47:12.863271 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1955,"thread_start_us":159,"threads_started":1}
I20260806 02:47:12.864300 19208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:12.848436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36342 (local address 127.1.183.131:45109)
0806 02:47:12.856387 (+ 7951us) server_negotiation.cc:206] Beginning negotiation
0806 02:47:12.856405 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:47:12.856441 (+ 36us) server_negotiation.cc:407] Connection header received
0806 02:47:12.856528 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:47:12.856533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:47:12.856780 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:47:12.856948 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:47:12.858642 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:12.859845 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:12.862197 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:12.863036 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:12.863152 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:12.863300 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 02:47:12.863999 (+ 699us) server_negotiation.cc:299] Negotiation successful
0806 02:47:12.864103 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7827,"thread_start_us":152,"threads_started":1}
I20260806 02:47:12.865592 19210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:12.845988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:38763 (local address 127.0.0.1:57262)
0806 02:47:12.851635 (+ 5647us) negotiation.cc:107] Waiting for socket to connect
0806 02:47:12.851704 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0806 02:47:12.851841 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:47:12.853650 (+ 1809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:47:12.853657 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:47:12.853680 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:47:12.853838 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:47:12.853844 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:47:12.857645 (+ 3801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:47:12.857648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:47:12.858853 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:47:12.858865 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:47:12.865212 (+ 6347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:47:12.865216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:47:12.865309 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:12.865332 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 02:47:12.865401 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 02:47:12.865468 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5443,"thread_start_us":276,"threads_started":1}
I20260806 02:47:12.866142 19206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:12.846151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57262 (local address 127.1.183.130:38763)
0806 02:47:12.852951 (+ 6800us) server_negotiation.cc:206] Beginning negotiation
0806 02:47:12.852969 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:47:12.852997 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:47:12.853082 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:47:12.853087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:47:12.853288 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:47:12.853466 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:47:12.856286 (+ 2820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:12.857494 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:12.860878 (+ 3384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:12.861707 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:12.864979 (+ 3272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:12.865148 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 02:47:12.865890 (+ 742us) server_negotiation.cc:299] Negotiation successful
0806 02:47:12.866005 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6672,"thread_start_us":96,"threads_started":1}
I20260806 02:47:12.869318 18856 tablet_service.cc:1467] Processing CreateTablet for tablet 6db431385adf406b857c6eba1bc24350 (DEFAULT_TABLE table=test-table [id=6dd2d32efe38412688adc8e7908150f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:47:12.869318 18685 tablet_service.cc:1467] Processing CreateTablet for tablet 6db431385adf406b857c6eba1bc24350 (DEFAULT_TABLE table=test-table [id=6dd2d32efe38412688adc8e7908150f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:47:12.870518 18685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6db431385adf406b857c6eba1bc24350. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:47:12.870566 18856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6db431385adf406b857c6eba1bc24350. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:47:12.864199 19209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:12.848800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:45109 (local address 127.0.0.1:36342)
0806 02:47:12.852318 (+ 3518us) negotiation.cc:107] Waiting for socket to connect
0806 02:47:12.852331 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:47:12.852417 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:47:12.858369 (+ 5952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:47:12.858373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:47:12.858393 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:47:12.858524 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:47:12.858529 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:47:12.860907 (+ 2378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:47:12.860911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:47:12.862058 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:47:12.862068 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:47:12.863725 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:47:12.863729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:47:12.863837 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:12.863862 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 02:47:12.863961 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 02:47:12.864036 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3458,"thread_start_us":182,"threads_started":1}
I20260806 02:47:12.884042 19073 tablet_service.cc:1467] Processing CreateTablet for tablet 6db431385adf406b857c6eba1bc24350 (DEFAULT_TABLE table=test-table [id=6dd2d32efe38412688adc8e7908150f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:47:12.885300 19073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6db431385adf406b857c6eba1bc24350. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:47:12.897111 19216 tablet_bootstrap.cc:492] T 6db431385adf406b857c6eba1bc24350 P aee1ab25738c45c3a85a8497dcca5547: Bootstrap starting.
I20260806 02:47:12.899264 19214 tablet_bootstrap.cc:492] T 6db431385adf406b857c6eba1bc24350 P e628a841cc774eaa859cc32345bb5fa2: Bootstrap starting.
I20260806 02:47:12.900468 19220 tablet_bootstrap.cc:492] T 6db431385adf406b857c6eba1bc24350 P e5afcb2024bf422492ccd28cbcf470c9: Bootstrap starting.
I20260806 02:47:12.902069 19214 tablet_bootstrap.cc:654] T 6db431385adf406b857c6eba1bc24350 P e628a841cc774eaa859cc32345bb5fa2: Neither blocks nor log segments found. Creating new log.
I20260806 02:47:12.902706 19216 tablet_bootstrap.cc:654] T 6db431385adf406b857c6eba1bc24350 P aee1ab25738c45c3a85a8497dcca5547: Neither blocks nor log segments found. Creating new log.
I20260806 02:47:12.903015 19214 log.cc:824] T 6db431385adf406b857c6eba1bc24350 P e628a841cc774eaa859cc32345bb5fa2: Log is configured to *not* fsync() on all Append() calls
I20260806 02:47:12.903813 19216 log.cc:824] T 6db431385adf406b857c6eba1bc24350 P aee1ab25738c45c3a85a8497dcca5547: Log is configured to *not* fsync() on all Append() calls
I20260806 02:47:12.906024 19220 tablet_bootstrap.cc:654] T 6db431385adf406b857c6eba1bc24350 P e5afcb2024bf422492ccd28cbcf470c9: Neither blocks nor log segments found. Creating new log.
I20260806 02:47:12.907574 19220 log.cc:824] T 6db431385adf406b857c6eba1bc24350 P e5afcb2024bf422492ccd28cbcf470c9: Log is configured to *not* fsync() on all Append() calls
I20260806 02:47:12.910385 19216 tablet_bootstrap.cc:492] T 6db431385adf406b857c6eba1bc24350 P aee1ab25738c45c3a85a8497dcca5547: No bootstrap required, opened a new log
I20260806 02:47:12.910805 19216 ts_tablet_manager.cc:1397] T 6db431385adf406b857c6eba1bc24350 P aee1ab25738c45c3a85a8497dcca5547: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260806 02:47:12.914036 19214 tablet_bootstrap.cc:492] T 6db431385adf406b857c6eba1bc24350 P e628a841cc774eaa859cc32345bb5fa2: No bootstrap required, opened a new log
I20260806 02:47:12.914283 19214 ts_tablet_manager.cc:1397] T 6db431385adf406b857c6eba1bc24350 P e628a841cc774eaa859cc32345bb5fa2: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20260806 02:47:12.917273 19216 raft_consensus.cc:348] T 6db431385adf406b857c6eba1bc24350 P aee1ab25738c45c3a85a8497dcca5547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e628a841cc774eaa859cc32345bb5fa2" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 38763 } } peers { permanent_uuid: "e5afcb2024bf422492ccd28cbcf470c9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45109 } } peers { permanent_uuid: "aee1ab25738c45c3a85a8497dcca5547" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 44451 } }
I20260806 02:47:12.917801 19216 raft_consensus.cc:374] T 6db431385adf406b857c6eba1bc24350 P aee1ab25738c45c3a85a8497dcca5547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:47:12.917912 19216 raft_consensus.cc:729] T 6db431385adf406b857c6eba1bc24350 P aee1ab25738c45c3a85a8497dcca5547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee1ab25738c45c3a85a8497dcca5547, State: Initialized, Role: FOLLOWER
I20260806 02:47:12.919186 19214 raft_consensus.cc:348] T 6db431385adf406b857c6eba1bc24350 P e628a841cc774eaa859cc32345bb5fa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e628a841cc774eaa859cc32345bb5fa2" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 38763 } } peers { permanent_uuid: "e5afcb2024bf422492ccd28cbcf470c9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45109 } } peers { permanent_uuid: "aee1ab25738c45c3a85a8497dcca5547" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 44451 } }
I20260806 02:47:12.919492 19214 raft_consensus.cc:374] T 6db431385adf406b857c6eba1bc24350 P e628a841cc774eaa859cc32345bb5fa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:47:12.919552 19214 raft_consensus.cc:729] T 6db431385adf406b857c6eba1bc24350 P e628a841cc774eaa859cc32345bb5fa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e628a841cc774eaa859cc32345bb5fa2, State: Initialized, Role: FOLLOWER
I20260806 02:47:12.920132 19216 consensus_queue.cc:260] T 6db431385adf406b857c6eba1bc24350 P aee1ab25738c45c3a85a8497dcca5547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e628a841cc774eaa859cc32345bb5fa2" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 38763 } } peers { permanent_uuid: "e5afcb2024bf422492ccd28cbcf470c9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45109 } } peers { permanent_uuid: "aee1ab25738c45c3a85a8497dcca5547" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 44451 } }
I20260806 02:47:12.920246 19214 consensus_queue.cc:260] T 6db431385adf406b857c6eba1bc24350 P e628a841cc774eaa859cc32345bb5fa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e628a841cc774eaa859cc32345bb5fa2" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 38763 } } peers { permanent_uuid: "e5afcb2024bf422492ccd28cbcf470c9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45109 } } peers { permanent_uuid: "aee1ab25738c45c3a85a8497dcca5547" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 44451 } }
I20260806 02:47:12.921496 19214 ts_tablet_manager.cc:1428] T 6db431385adf406b857c6eba1bc24350 P e628a841cc774eaa859cc32345bb5fa2: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260806 02:47:12.921921 18975 heartbeater.cc:503] Master 127.1.183.190:33895 was elected leader, sending a full tablet report...
I20260806 02:47:12.923878 19220 tablet_bootstrap.cc:492] T 6db431385adf406b857c6eba1bc24350 P e5afcb2024bf422492ccd28cbcf470c9: No bootstrap required, opened a new log
I20260806 02:47:12.924106 19220 ts_tablet_manager.cc:1397] T 6db431385adf406b857c6eba1bc24350 P e5afcb2024bf422492ccd28cbcf470c9: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s
I20260806 02:47:12.930016 19220 raft_consensus.cc:348] T 6db431385adf406b857c6eba1bc24350 P e5afcb2024bf422492ccd28cbcf470c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e628a841cc774eaa859cc32345bb5fa2" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 38763 } } peers { permanent_uuid: "e5afcb2024bf422492ccd28cbcf470c9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45109 } } peers { permanent_uuid: "aee1ab25738c45c3a85a8497dcca5547" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 44451 } }
I20260806 02:47:12.930392 19220 raft_consensus.cc:374] T 6db431385adf406b857c6eba1bc24350 P e5afcb2024bf422492ccd28cbcf470c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:47:12.930475 19220 raft_consensus.cc:729] T 6db431385adf406b857c6eba1bc24350 P e5afcb2024bf422492ccd28cbcf470c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5afcb2024bf422492ccd28cbcf470c9, State: Initialized, Role: FOLLOWER
I20260806 02:47:12.931008 19216 ts_tablet_manager.cc:1428] T 6db431385adf406b857c6eba1bc24350 P aee1ab25738c45c3a85a8497dcca5547: Time spent starting tablet: real 0.020s user 0.000s sys 0.007s
I20260806 02:47:12.931457 18775 heartbeater.cc:503] Master 127.1.183.190:33895 was elected leader, sending a full tablet report...
I20260806 02:47:12.931761 19220 consensus_queue.cc:260] T 6db431385adf406b857c6eba1bc24350 P e5afcb2024bf422492ccd28cbcf470c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e628a841cc774eaa859cc32345bb5fa2" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 38763 } } peers { permanent_uuid: "e5afcb2024bf422492ccd28cbcf470c9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45109 } } peers { permanent_uuid: "aee1ab25738c45c3a85a8497dcca5547" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 44451 } }
I20260806 02:47:12.933149 19220 ts_tablet_manager.cc:1428] T 6db431385adf406b857c6eba1bc24350 P e5afcb2024bf422492ccd28cbcf470c9: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20260806 02:47:12.933554 19176 heartbeater.cc:503] Master 127.1.183.190:33895 was elected leader, sending a full tablet report...
W20260806 02:47:12.970682 18779 tablet.cc:2369] T 6db431385adf406b857c6eba1bc24350 P aee1ab25738c45c3a85a8497dcca5547: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 02:47:12 dist-test-slave-gmjp krb5kdc[17741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984432, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for HTTP/127.1.183.148@KRBTEST.COM
W20260806 02:47:13.015767 19177 tablet.cc:2369] T 6db431385adf406b857c6eba1bc24350 P e5afcb2024bf422492ccd28cbcf470c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:47:13.080492 18978 tablet.cc:2369] T 6db431385adf406b857c6eba1bc24350 P e628a841cc774eaa859cc32345bb5fa2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:47:13.212906 19227 raft_consensus.cc:482] T 6db431385adf406b857c6eba1bc24350 P aee1ab25738c45c3a85a8497dcca5547 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:47:13.213171 19227 raft_consensus.cc:504] T 6db431385adf406b857c6eba1bc24350 P aee1ab25738c45c3a85a8497dcca5547 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e628a841cc774eaa859cc32345bb5fa2" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 38763 } } peers { permanent_uuid: "e5afcb2024bf422492ccd28cbcf470c9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45109 } } peers { permanent_uuid: "aee1ab25738c45c3a85a8497dcca5547" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 44451 } }
I20260806 02:47:13.214514 19227 leader_election.cc:288] T 6db431385adf406b857c6eba1bc24350 P aee1ab25738c45c3a85a8497dcca5547 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e628a841cc774eaa859cc32345bb5fa2 (127.1.183.130:38763), e5afcb2024bf422492ccd28cbcf470c9 (127.1.183.131:45109)
I20260806 02:47:13.220793 19206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:13.215252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:60955 (local address 127.1.183.130:38763)
0806 02:47:13.215413 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 02:47:13.215418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:47:13.215560 (+ 142us) server_negotiation.cc:407] Connection header received
0806 02:47:13.215793 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:47:13.215797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:47:13.215848 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:47:13.215900 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:47:13.216385 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:13.217188 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:13.219279 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:13.219924 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:13.220027 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:13.220086 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 02:47:13.220524 (+ 438us) server_negotiation.cc:299] Negotiation successful
0806 02:47:13.220614 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:47:13.221356 19208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:13.215233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:60031 (local address 127.1.183.131:45109)
0806 02:47:13.215376 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 02:47:13.215382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:47:13.215633 (+ 251us) server_negotiation.cc:407] Connection header received
0806 02:47:13.215815 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:47:13.215818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:47:13.215871 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:47:13.215922 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:47:13.216400 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:13.217188 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:13.219931 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:13.220593 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:13.220698 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:13.220752 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 02:47:13.221149 (+ 397us) server_negotiation.cc:299] Negotiation successful
0806 02:47:13.221241 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:47:13.221838 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:13.214819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:45109 (local address 127.1.183.129:60031)
0806 02:47:13.215552 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0806 02:47:13.215565 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:47:13.215691 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:47:13.216075 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:47:13.216078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:47:13.216144 (+ 66us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:47:13.216267 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:47:13.216273 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:47:13.217357 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:47:13.217360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:47:13.219112 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:47:13.219126 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:47:13.220881 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:47:13.220884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:47:13.220989 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:13.221014 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 02:47:13.221563 (+ 549us) client_negotiation.cc:240] Negotiation successful
0806 02:47:13.221668 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":147,"spinlock_wait_cycles":22912,"thread_start_us":82,"threads_started":1}
I20260806 02:47:13.222594 19079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6db431385adf406b857c6eba1bc24350" candidate_uuid: "aee1ab25738c45c3a85a8497dcca5547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5afcb2024bf422492ccd28cbcf470c9" is_pre_election: true
I20260806 02:47:13.222998 19079 raft_consensus.cc:2454] T 6db431385adf406b857c6eba1bc24350 P e5afcb2024bf422492ccd28cbcf470c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aee1ab25738c45c3a85a8497dcca5547 in term 0.
I20260806 02:47:13.223606 19234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:13.214687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:38763 (local address 127.1.183.129:60955)
0806 02:47:13.215465 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0806 02:47:13.215482 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:47:13.215654 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:47:13.216075 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:47:13.216078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:47:13.216099 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:47:13.216239 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:47:13.216249 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:47:13.217357 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:47:13.217360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:47:13.219112 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:47:13.219126 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:47:13.220219 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:47:13.220223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:47:13.220316 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:13.220398 (+ 82us) client_negotiation.cc:769] Sending connection context
0806 02:47:13.221763 (+ 1365us) client_negotiation.cc:240] Negotiation successful
0806 02:47:13.221834 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":198,"thread_start_us":239,"threads_started":1}
I20260806 02:47:13.223695 18608 leader_election.cc:302] T 6db431385adf406b857c6eba1bc24350 P aee1ab25738c45c3a85a8497dcca5547 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aee1ab25738c45c3a85a8497dcca5547, e5afcb2024bf422492ccd28cbcf470c9; no voters:
I20260806 02:47:13.224049 19227 raft_consensus.cc:2781] T 6db431385adf406b857c6eba1bc24350 P aee1ab25738c45c3a85a8497dcca5547 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:47:13.224144 19227 raft_consensus.cc:482] T 6db431385adf406b857c6eba1bc24350 P aee1ab25738c45c3a85a8497dcca5547 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:47:13.224251 19227 raft_consensus.cc:3037] T 6db431385adf406b857c6eba1bc24350 P aee1ab25738c45c3a85a8497dcca5547 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:47:13.224402 18906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6db431385adf406b857c6eba1bc24350" candidate_uuid: "aee1ab25738c45c3a85a8497dcca5547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e628a841cc774eaa859cc32345bb5fa2" is_pre_election: true
I20260806 02:47:13.224828 18906 raft_consensus.cc:2454] T 6db431385adf406b857c6eba1bc24350 P e628a841cc774eaa859cc32345bb5fa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aee1ab25738c45c3a85a8497dcca5547 in term 0.
I20260806 02:47:13.225425 19227 raft_consensus.cc:504] T 6db431385adf406b857c6eba1bc24350 P aee1ab25738c45c3a85a8497dcca5547 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e628a841cc774eaa859cc32345bb5fa2" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 38763 } } peers { permanent_uuid: "e5afcb2024bf422492ccd28cbcf470c9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45109 } } peers { permanent_uuid: "aee1ab25738c45c3a85a8497dcca5547" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 44451 } }
I20260806 02:47:13.225847 19227 leader_election.cc:288] T 6db431385adf406b857c6eba1bc24350 P aee1ab25738c45c3a85a8497dcca5547 [CANDIDATE]: Term 1 election: Requested vote from peers e628a841cc774eaa859cc32345bb5fa2 (127.1.183.130:38763), e5afcb2024bf422492ccd28cbcf470c9 (127.1.183.131:45109)
I20260806 02:47:13.226315 18906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6db431385adf406b857c6eba1bc24350" candidate_uuid: "aee1ab25738c45c3a85a8497dcca5547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e628a841cc774eaa859cc32345bb5fa2"
I20260806 02:47:13.226477 18906 raft_consensus.cc:3037] T 6db431385adf406b857c6eba1bc24350 P e628a841cc774eaa859cc32345bb5fa2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:47:13.226454 19079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6db431385adf406b857c6eba1bc24350" candidate_uuid: "aee1ab25738c45c3a85a8497dcca5547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5afcb2024bf422492ccd28cbcf470c9"
I20260806 02:47:13.226639 19079 raft_consensus.cc:3037] T 6db431385adf406b857c6eba1bc24350 P e5afcb2024bf422492ccd28cbcf470c9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:47:13.227779 18906 raft_consensus.cc:2454] T 6db431385adf406b857c6eba1bc24350 P e628a841cc774eaa859cc32345bb5fa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aee1ab25738c45c3a85a8497dcca5547 in term 1.
I20260806 02:47:13.228339 19079 raft_consensus.cc:2454] T 6db431385adf406b857c6eba1bc24350 P e5afcb2024bf422492ccd28cbcf470c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aee1ab25738c45c3a85a8497dcca5547 in term 1.
I20260806 02:47:13.228984 18609 leader_election.cc:302] T 6db431385adf406b857c6eba1bc24350 P aee1ab25738c45c3a85a8497dcca5547 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aee1ab25738c45c3a85a8497dcca5547, e628a841cc774eaa859cc32345bb5fa2; no voters:
I20260806 02:47:13.229193 19227 raft_consensus.cc:2781] T 6db431385adf406b857c6eba1bc24350 P aee1ab25738c45c3a85a8497dcca5547 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:47:13.229519 19227 raft_consensus.cc:686] T 6db431385adf406b857c6eba1bc24350 P aee1ab25738c45c3a85a8497dcca5547 [term 1 LEADER]: Becoming Leader. State: Replica: aee1ab25738c45c3a85a8497dcca5547, State: Running, Role: LEADER
I20260806 02:47:13.229961 19227 consensus_queue.cc:237] T 6db431385adf406b857c6eba1bc24350 P aee1ab25738c45c3a85a8497dcca5547 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e628a841cc774eaa859cc32345bb5fa2" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 38763 } } peers { permanent_uuid: "e5afcb2024bf422492ccd28cbcf470c9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45109 } } peers { permanent_uuid: "aee1ab25738c45c3a85a8497dcca5547" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 44451 } }
I20260806 02:47:13.233625 18451 catalog_manager.cc:5495] T 6db431385adf406b857c6eba1bc24350 P aee1ab25738c45c3a85a8497dcca5547 reported cstate change: term changed from 0 to 1, leader changed from <none> to aee1ab25738c45c3a85a8497dcca5547 (127.1.183.129). New cstate: current_term: 1 leader_uuid: "aee1ab25738c45c3a85a8497dcca5547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e628a841cc774eaa859cc32345bb5fa2" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 38763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5afcb2024bf422492ccd28cbcf470c9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aee1ab25738c45c3a85a8497dcca5547" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 44451 } health_report { overall_health: HEALTHY } } }
I20260806 02:47:13.310842 19212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:13.306821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50318 (local address 127.1.183.129:44451)
0806 02:47:13.306987 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 02:47:13.306995 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:47:13.307011 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:47:13.307105 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:47:13.307108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:47:13.307176 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:47:13.307297 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:47:13.307584 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:13.308101 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:13.309127 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:47:13.309367 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:47:13.309504 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:47:13.309839 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:47:13.310382 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:47:13.310554 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 02:47:13.310588 (+ 34us) server_negotiation.cc:299] Negotiation successful
0806 02:47:13.310656 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 02:47:13.317628 19079 raft_consensus.cc:1261] T 6db431385adf406b857c6eba1bc24350 P e5afcb2024bf422492ccd28cbcf470c9 [term 1 FOLLOWER]: Refusing update from remote peer aee1ab25738c45c3a85a8497dcca5547: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:47:13.317651 18906 raft_consensus.cc:1261] T 6db431385adf406b857c6eba1bc24350 P e628a841cc774eaa859cc32345bb5fa2 [term 1 FOLLOWER]: Refusing update from remote peer aee1ab25738c45c3a85a8497dcca5547: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:47:13.318430 19246 consensus_queue.cc:1035] T 6db431385adf406b857c6eba1bc24350 P aee1ab25738c45c3a85a8497dcca5547 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5afcb2024bf422492ccd28cbcf470c9" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:47:13.318656 19227 consensus_queue.cc:1035] T 6db431385adf406b857c6eba1bc24350 P aee1ab25738c45c3a85a8497dcca5547 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e628a841cc774eaa859cc32345bb5fa2" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 38763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:47:13.328771 19253 mvcc.cc:205] Tried to move back new op lower bound from 7315392238860210176 to 7315392238513082368. Current Snapshot: MvccSnapshot[applied={T|T < 7315392238860210176}]
I20260806 02:47:13.338416 18451 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:47:13.338631 18451 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:47:13.341588 18451 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ba889e2cbfd74dcab01bc48a39d72338: Sending DeleteTablet for 3 replicas of tablet 6db431385adf406b857c6eba1bc24350
I20260806 02:47:13.342501 19073 tablet_service.cc:1514] Processing DeleteTablet for tablet 6db431385adf406b857c6eba1bc24350 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:47:13 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:36342
I20260806 02:47:13.342628 18856 tablet_service.cc:1514] Processing DeleteTablet for tablet 6db431385adf406b857c6eba1bc24350 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:47:13 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:57262
I20260806 02:47:13.342676 18685 tablet_service.cc:1514] Processing DeleteTablet for tablet 6db431385adf406b857c6eba1bc24350 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:47:13 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:50312
I20260806 02:47:13.343165 19264 tablet_replica.cc:330] T 6db431385adf406b857c6eba1bc24350 P e5afcb2024bf422492ccd28cbcf470c9: stopping tablet replica
I20260806 02:47:13.343549 19264 raft_consensus.cc:2229] T 6db431385adf406b857c6eba1bc24350 P e5afcb2024bf422492ccd28cbcf470c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:47:13.343866 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 18578
I20260806 02:47:13.343918 19264 raft_consensus.cc:2258] T 6db431385adf406b857c6eba1bc24350 P e5afcb2024bf422492ccd28cbcf470c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:47:13.352031 19264 ts_tablet_manager.cc:1905] T 6db431385adf406b857c6eba1bc24350 P e5afcb2024bf422492ccd28cbcf470c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:47:13.355485 19264 ts_tablet_manager.cc:1918] T 6db431385adf406b857c6eba1bc24350 P e5afcb2024bf422492ccd28cbcf470c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:47:13.355615 19264 log.cc:1196] T 6db431385adf406b857c6eba1bc24350 P e5afcb2024bf422492ccd28cbcf470c9: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-2/wal/wals/6db431385adf406b857c6eba1bc24350
I20260806 02:47:13.355950 19264 ts_tablet_manager.cc:1939] T 6db431385adf406b857c6eba1bc24350 P e5afcb2024bf422492ccd28cbcf470c9: Deleting consensus metadata
I20260806 02:47:13.356478 19266 tablet_replica.cc:330] T 6db431385adf406b857c6eba1bc24350 P e628a841cc774eaa859cc32345bb5fa2: stopping tablet replica
I20260806 02:47:13.357519 18429 catalog_manager.cc:4841] TS e5afcb2024bf422492ccd28cbcf470c9 (127.1.183.131:45109): tablet 6db431385adf406b857c6eba1bc24350 (table test-table [id=6dd2d32efe38412688adc8e7908150f8]) successfully deleted
I20260806 02:47:13.358080 19266 raft_consensus.cc:2229] T 6db431385adf406b857c6eba1bc24350 P e628a841cc774eaa859cc32345bb5fa2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:47:13.358506 19266 raft_consensus.cc:2258] T 6db431385adf406b857c6eba1bc24350 P e628a841cc774eaa859cc32345bb5fa2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:47:13.361555 19266 ts_tablet_manager.cc:1905] T 6db431385adf406b857c6eba1bc24350 P e628a841cc774eaa859cc32345bb5fa2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:47:13.363965 19266 ts_tablet_manager.cc:1918] T 6db431385adf406b857c6eba1bc24350 P e628a841cc774eaa859cc32345bb5fa2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:47:13.364099 19266 log.cc:1196] T 6db431385adf406b857c6eba1bc24350 P e628a841cc774eaa859cc32345bb5fa2: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984313887056-1758-0/minicluster-data/ts-1/wal/wals/6db431385adf406b857c6eba1bc24350
I20260806 02:47:13.364455 19266 ts_tablet_manager.cc:1939] T 6db431385adf406b857c6eba1bc24350 P e628a841cc774eaa859cc32345bb5fa2: Deleting consensus metadata
I20260806 02:47:13.365808 18430 catalog_manager.cc:4841] TS e628a841cc774eaa859cc32345bb5fa2 (127.1.183.130:38763): tablet 6db431385adf406b857c6eba1bc24350 (table test-table [id=6dd2d32efe38412688adc8e7908150f8]) successfully deleted
W20260806 02:47:13.366516 18428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.183.129:44451 (error 108)
I20260806 02:47:13.366772 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 18786
I20260806 02:47:13.376015 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:47:13.375552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:44451 (local address 127.0.0.1:50320)
0806 02:47:13.375782 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0806 02:47:13.375876 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.129:44451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260806 02:47:13.376343 18428 catalog_manager.cc:4584] TS aee1ab25738c45c3a85a8497dcca5547 (127.1.183.129:44451): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6db431385adf406b857c6eba1bc24350: Network error: Client connection negotiation failed: client connection to 127.1.183.129:44451: connect: Connection refused (error 111)
I20260806 02:47:13.383170 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 18982
I20260806 02:47:13.403029 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 18402
I20260806 02:47:13.422370 1758 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 02:47:13.769757 1758 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 02:47:13.769956 1758 mini_ranger.cc:66] Stopping Ranger...
I20260806 02:47:14.127329 1758 mini_ranger.cc:68] Stopped Ranger
2026-08-06 02:47:14.127 UTC [17782] LOG: received smart shutdown request
2026-08-06 02:47:14.131 UTC [17782] LOG: background worker "logical replication launcher" (PID 17792) exited with exit code 1
2026-08-06 02:47:14.131 UTC [17786] LOG: shutting down
2026-08-06 02:47:14.309 UTC [17782] LOG: database system is shut down
2026-08-06T02:47:14Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50537 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:47:14 dist-test-slave-gmjp krb5kdc[19279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:47:14 dist-test-slave-gmjp krb5kdc[19279](info): set up 2 sockets
Aug 06 02:47:14 dist-test-slave-gmjp krb5kdc[19279](info): commencing operation
krb5kdc: starting...
W20260806 02:47:15.664963 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.213s user 0.000s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:47:15 dist-test-slave-gmjp krb5kdc[19279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:47:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:47:15Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.183.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.183.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.183.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/rangeradmin_127.1.183.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.183.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/rangeradmin_127.1.183.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.183.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.183.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.183.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/rangerlookup_127.1.183.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.183.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/rangerlookup_127.1.183.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.183.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.183.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.183.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/HTTP_127.1.183.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.183.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/HTTP_127.1.183.148@KRBTEST.COM.keytab.
I20260806 02:47:15.853323 1758 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task0hWVsi/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/postgres -l logfile start
2026-08-06 02:47:18.421 UTC [19325] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 02:47:18.421 UTC [19325] LOG: listening on IPv4 address "127.1.183.148", port 45261
2026-08-06 02:47:18.425 UTC [19325] LOG: listening on Unix socket "/tmp/.s.PGSQL.45261"
2026-08-06 02:47:18.447 UTC [19330] LOG: database system was shut down at 2026-08-06 02:47:16 UTC
2026-08-06 02:47:18.451 UTC [19325] LOG: database system is ready to accept connections
I20260806 02:47:19.584761 1758 mini_postgres.cc:96] Postgres bound to 45261
2026-08-06 02:47:19.592 UTC [19345] FATAL: database "slave" does not exist
127.1.183.148:45261 - accepting connections
I20260806 02:47:19.593395 1758 mini_ranger.cc:161] Starting Ranger...
I20260806 02:47:19.611209 1758 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 02:47:20.126839 1758 mini_ranger.cc:87] Created ranger Postgres database
I20260806 02:47:20.126940 1758 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ranger-admin
2026-08-06 02:47:20,433 [I] DB FLAVOR :POSTGRES
2026-08-06 02:47:20,434 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 02:47:20,434 [I] Checking connection..
2026-08-06 02:47:20,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:45261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 02:47:20,708 [I] Checking connection passed.
2026-08-06 02:47:20,708 [I] --------- Verifying version history table ---------
2026-08-06 02:47:20,709 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:45261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:47:20,970 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 02:47:20,970 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:45261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:47:21,261 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 02:47:21,262 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 02:47:21,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:45261/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 02:47:21.499 UTC [19438] WARNING: there is no transaction in progress
2026-08-06 02:47:21,508 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 02:47:21,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:45261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:47:21,772 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 02:47:21,773 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 02:47:21,773 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:45261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 02:47:22,054 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:45261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 02:47:22,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:45261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gmjp.c.gcp-upstream.internal','N') ;"
2026-08-06 02:47:22,584 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:45261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 02:47:22,860 [I] Table x_portal_user does not exist in database ranger
2026-08-06 02:47:22,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:45261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 02:47:23,138 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 02:47:23,139 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:45261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 02:47:23,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:45261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 02:47:23,668 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 02:47:23,669 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:45261/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 02:47:24.668 UTC [19611] WARNING: there is no transaction in progress
2026-08-06 02:47:24.685 UTC [19611] WARNING: there is no transaction in progress
2026-08-06 02:47:24.702 UTC [19611] WARNING: there is no transaction in progress
2026-08-06 02:47:24.719 UTC [19611] WARNING: there is no transaction in progress
2026-08-06 02:47:24.738 UTC [19611] WARNING: there is no transaction in progress
2026-08-06 02:47:24.760 UTC [19611] WARNING: there is no transaction in progress
2026-08-06 02:47:24.819 UTC [19611] WARNING: there is no transaction in progress
2026-08-06 02:47:24.826 UTC [19611] WARNING: there is no transaction in progress
2026-08-06 02:47:24.843 UTC [19611] WARNING: there is no transaction in progress
2026-08-06 02:47:24.854 UTC [19611] WARNING: there is no transaction in progress
2026-08-06 02:47:24.866 UTC [19611] WARNING: there is no transaction in progress
2026-08-06 02:47:24.881 UTC [19611] WARNING: there is no transaction in progress
2026-08-06 02:47:24.891 UTC [19611] WARNING: there is no transaction in progress
2026-08-06 02:47:24.898 UTC [19611] WARNING: there is no transaction in progress
2026-08-06 02:47:24.905 UTC [19611] WARNING: there is no transaction in progress
2026-08-06 02:47:24.912 UTC [19611] WARNING: there is no transaction in progress
2026-08-06 02:47:25,597 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 02:47:25,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:45261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 02:47:25,866 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 02:47:25,866 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:45261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 02:47:26,139 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 02:47:26,140 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:45261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 02:47:26,400 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 02:47:26,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:45261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 02:47:26,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:45261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 02:47:26,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:45261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gmjp.c.gcp-upstream.internal';"
2026-08-06 02:47:27,215 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 02:47:27,216 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:45261/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 02:47:27,482 [I] DB_PATCHES have already been applied
I20260806 02:47:27.488432 1758 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ranger-admin:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task0hWVsi/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 02:47:27.488512 1758 mini_ranger.cc:193] Using host: 127.1.183.148
I20260806 02:47:27.493281 1758 mini_ranger.cc:227] Ranger admin URL: http://127.1.183.148:55111
Aug 06, 2026 2:47:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 2:47:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 2:47:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 2:47:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 2:47:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 2:47:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 2:47:28 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 02:47:28.819052 1758 mini_ranger.cc:160] Time spent starting Ranger: real 9.226s user 0.000s sys 0.013s
Aug 06, 2026 2:47:49 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 2:47:49 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 06, 2026 2:47:49 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 2:47:49 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 2:47:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 2:47:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 2:47:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 2:47:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 2:47:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 2:47:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 2:47:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 2:47:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 2:47:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 2:47:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 2:47:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 2:47:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 2:47:49 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 2:47:50 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 02:47:50.798836 1758 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.183.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.183.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.183.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/rangerkms_127.1.183.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.183.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/rangerkms_127.1.183.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.183.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.183.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.183.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/HTTP_127.1.183.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.183.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/HTTP_127.1.183.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 02:47:50 dist-test-slave-gmjp krb5kdc[19279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984470, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 02:47:50.893749 1758 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 02:47:50.905190 1758 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 02:47:51.853103 1758 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 02:47:51.853251 1758 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ranger-kms
I20260806 02:47:51.853279 1758 mini_ranger_kms.cc:227] Using postgres at 127.1.183.148:45261
2026-08-06 02:47:52,054 [I] DB FLAVOR :POSTGRES
2026-08-06 02:47:52,054 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 02:47:52,054 [I] Checking connection
2026-08-06 02:47:52,054 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:45261/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 02:47:52,331 [I] connection success
2026-08-06 02:47:52,331 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 02:47:52,331 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 02:47:52,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:45261/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 02:47:52,579 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 02:47:52,580 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 02:47:52,580 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 02:47:52,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.183.148:45261/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 02:47:52,873 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 02:47:53.377483 1758 mini_ranger_kms.cc:310] Created kms service
I20260806 02:47:53.590616 1758 mini_ranger_kms.cc:326] Created kudu user
I20260806 02:47:53.637410 1758 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 02:47:53 dist-test-slave-gmjp krb5kdc[19279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984470, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.183.148@KRBTEST.COM
I20260806 02:47:54.061721 1758 mini_ranger_kms.cc:384] Added ranger policy
I20260806 02:47:54.061884 1758 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ranger-kms:/tmp/dist-test-task0hWVsi/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task0hWVsi/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 02:47:54.061904 1758 mini_ranger_kms.cc:240] Using host: 127.1.183.148
I20260806 02:47:54.066870 1758 mini_ranger_kms.cc:276] Ranger KMS PID: 19882
I20260806 02:47:54.066969 1758 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.183.148:53643
Aug 06, 2026 2:47:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 2:47:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 2:47:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 2:47:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 2:47:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 2:47:55 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 02:47:56 dist-test-slave-gmjp krb5kdc[19279](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785984476, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.183.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:47:56.810034 1758 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.916s user 0.015s sys 0.171s
I20260806 02:47:56.810143 1758 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 02:47:56.810200 1758 mini_ranger_kms.cc:401] 127.1.183.148:53643/kms/v1/keys
Aug 06 02:47:57 dist-test-slave-gmjp krb5kdc[19279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984476, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.183.148@KRBTEST.COM for HTTP/127.1.183.148@KRBTEST.COM
Aug 06 02:47:57 dist-test-slave-gmjp krb5kdc[19279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984476, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.183.148@KRBTEST.COM for HTTP/127.1.183.148@KRBTEST.COM
Aug 06 02:47:59 dist-test-slave-gmjp krb5kdc[19279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
I20260806 02:47:59.977571 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:35991
--webserver_interface=127.1.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:33657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:35991
--ranger_config_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.183.148:53643/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:48:00.099793 19966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:00.100167 19966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:00.100292 19966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:00.104494 19966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:48:00.104600 19966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:00.104638 19966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:00.104700 19966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:48:00.104755 19966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:48:00.110785 19966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:33657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.183.148:53643/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:35991
--ranger_config_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:35991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:48:00.112048 19966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:00.113216 19966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:00.123113 19974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:00.123345 19971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:00.123152 19972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:00.123313 19966 server_base.cc:1034] running on GCE node
I20260806 02:48:00.124332 19966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:00.126092 19966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:00.127326 19966 hybrid_clock.cc:648] HybridClock initialized: now 1785984480127288 us; error 55 us; skew 500 ppm
Aug 06 02:48:00 dist-test-slave-gmjp krb5kdc[19279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984480, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:00.135092 19966 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:48:00.136464 19966 webserver.cc:460] Webserver started at http://127.1.183.190:38637/ using document root <none> and password file <none>
I20260806 02:48:00.137104 19966 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:00.137198 19966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:00.137439 19966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:48:00 dist-test-slave-gmjp krb5kdc[19279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984480, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for HTTP/127.1.183.148@KRBTEST.COM
I20260806 02:48:00.191747 19966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "5275dd1bee554875a224893a748eb006"
format_stamp: "Formatted at 2026-08-06 02:48:00 on dist-test-slave-gmjp"
server_key: "dc27699b094e428dac4e52acbbdf1b4d"
server_key_iv: "0643c6c4ced5cbfcde35fb114742aa8b"
server_key_version: "kuduclusterkey@0"
I20260806 02:48:00.192639 19966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "5275dd1bee554875a224893a748eb006"
format_stamp: "Formatted at 2026-08-06 02:48:00 on dist-test-slave-gmjp"
server_key: "dc27699b094e428dac4e52acbbdf1b4d"
server_key_iv: "0643c6c4ced5cbfcde35fb114742aa8b"
server_key_version: "kuduclusterkey@0"
I20260806 02:48:00.197140 19966 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:48:00.212749 19982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:00.214275 19966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:48:00.214444 19966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "5275dd1bee554875a224893a748eb006"
format_stamp: "Formatted at 2026-08-06 02:48:00 on dist-test-slave-gmjp"
server_key: "dc27699b094e428dac4e52acbbdf1b4d"
server_key_iv: "0643c6c4ced5cbfcde35fb114742aa8b"
server_key_version: "kuduclusterkey@0"
I20260806 02:48:00.214617 19966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:00.228899 19966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:00.232347 19966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:00.232577 19966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:00.241396 19966 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:35991
I20260806 02:48:00.241398 20063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:35991 every 8 connection(s)
I20260806 02:48:00.242463 19966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:48:00.245536 20064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:48:00.247895 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 19966
I20260806 02:48:00.248030 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:48:00.248392 1758 external_mini_cluster.cc:1428] Setting key f60d43b1236468a78664788691f53167
I20260806 02:48:00.252383 20064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006: Bootstrap starting.
I20260806 02:48:00.254947 20064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:00.255952 20064 log.cc:824] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:00.258559 20064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006: No bootstrap required, opened a new log
Aug 06 02:48:00 dist-test-slave-gmjp krb5kdc[19279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:48:00.263036 20064 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5275dd1bee554875a224893a748eb006" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35991 } }
I20260806 02:48:00.263381 20064 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:00.263447 20064 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5275dd1bee554875a224893a748eb006, State: Initialized, Role: FOLLOWER
I20260806 02:48:00.264004 20064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5275dd1bee554875a224893a748eb006" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35991 } }
I20260806 02:48:00.264215 20064 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:48:00.264297 20064 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:48:00.264396 20064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:00.265071 20072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:00.249979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58286 (local address 127.1.183.190:35991)
0806 02:48:00.250499 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:00.250525 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:00.250573 (+ 48us) server_negotiation.cc:407] Connection header received
0806 02:48:00.251428 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:00.251452 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:00.251941 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:00.252359 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:00.253310 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:00.254884 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:00.256214 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:00.256829 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:00.257028 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:00.259797 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:00.259803 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:00.259818 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:00.259847 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:00.261597 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:00.262080 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:00.262091 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:00.262096 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:00.262205 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:00.262507 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:00.262511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:00.262513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:00.262941 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:00.263167 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:00.263899 (+ 732us) server_negotiation.cc:299] Negotiation successful
0806 02:48:00.264168 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":351,"thread_start_us":158,"threads_started":1}
I20260806 02:48:00.265949 20064 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5275dd1bee554875a224893a748eb006" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35991 } }
I20260806 02:48:00.266366 20064 leader_election.cc:302] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5275dd1bee554875a224893a748eb006; no voters:
I20260806 02:48:00.266729 20064 leader_election.cc:288] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:48:00.266914 20074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:48:00.267174 20074 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [term 1 LEADER]: Becoming Leader. State: Replica: 5275dd1bee554875a224893a748eb006, State: Running, Role: LEADER
I20260806 02:48:00.267884 20074 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5275dd1bee554875a224893a748eb006" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35991 } }
I20260806 02:48:00.268190 20064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:48:00.269402 20079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5275dd1bee554875a224893a748eb006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5275dd1bee554875a224893a748eb006" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35991 } } }
I20260806 02:48:00.269536 20079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [sys.catalog]: This master's current role is: LEADER
I20260806 02:48:00.270222 20076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5275dd1bee554875a224893a748eb006. Latest consensus state: current_term: 1 leader_uuid: "5275dd1bee554875a224893a748eb006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5275dd1bee554875a224893a748eb006" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35991 } } }
I20260806 02:48:00.270336 20076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [sys.catalog]: This master's current role is: LEADER
I20260806 02:48:00.271304 20082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:48:00.275352 20082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:48:00.275869 20064 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 02:48:00.286690 20082 catalog_manager.cc:1347] Generated new cluster ID: f28279f310e344958b7a1329941742e7
I20260806 02:48:00.286860 20082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:48:00.298570 20082 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:48:00.299943 20082 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:48:00.305486 20082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006: Generated new TSK 0
I20260806 02:48:00.306430 20082 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
I20260806 02:48:01.151682 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.183.148:53643/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.129:0
--local_ip_for_outbound_sockets=127.1.183.129
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:35991
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:33657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:48:01.333688 20146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:01.334017 20146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:01.334144 20146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:01.338802 20146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:01.338907 20146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:01.339021 20146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.129
I20260806 02:48:01.345932 20146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:33657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.183.148:53643/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:35991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.129
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:48:01.347259 20146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:01.348548 20146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:01.360514 20158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:01.360965 20157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:01.361339 20162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:01.361552 20146 server_base.cc:1034] running on GCE node
I20260806 02:48:01.362082 20146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:01.362983 20146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:01.364301 20146 hybrid_clock.cc:648] HybridClock initialized: now 1785984481364246 us; error 60 us; skew 500 ppm
Aug 06 02:48:01 dist-test-slave-gmjp krb5kdc[19279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984481, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:01.368005 20146 init.cc:377] Logged in from keytab as kudu/127.1.183.129@KRBTEST.COM (short username kudu)
I20260806 02:48:01.369086 20146 webserver.cc:460] Webserver started at http://127.1.183.129:40471/ using document root <none> and password file <none>
I20260806 02:48:01.369870 20146 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:01.369979 20146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:01.370333 20146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:48:01 dist-test-slave-gmjp krb5kdc[19279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984481, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for HTTP/127.1.183.148@KRBTEST.COM
I20260806 02:48:01.389582 20146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-0/data/instance:
uuid: "0eb80f8bbbe64dd7a94680e19017f536"
format_stamp: "Formatted at 2026-08-06 02:48:01 on dist-test-slave-gmjp"
server_key: "7cffed46b1cb168818e63483a7d04e75"
server_key_iv: "8b006a8711fe2acda41cd08a25e11765"
server_key_version: "kuduclusterkey@0"
I20260806 02:48:01.390571 20146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance:
uuid: "0eb80f8bbbe64dd7a94680e19017f536"
format_stamp: "Formatted at 2026-08-06 02:48:01 on dist-test-slave-gmjp"
server_key: "7cffed46b1cb168818e63483a7d04e75"
server_key_iv: "8b006a8711fe2acda41cd08a25e11765"
server_key_version: "kuduclusterkey@0"
I20260806 02:48:01.401284 20146 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.002s sys 0.005s
I20260806 02:48:01.429720 20173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:01.431732 20146 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.002s sys 0.000s
I20260806 02:48:01.431915 20146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-0/wal
uuid: "0eb80f8bbbe64dd7a94680e19017f536"
format_stamp: "Formatted at 2026-08-06 02:48:01 on dist-test-slave-gmjp"
server_key: "7cffed46b1cb168818e63483a7d04e75"
server_key_iv: "8b006a8711fe2acda41cd08a25e11765"
server_key_version: "kuduclusterkey@0"
I20260806 02:48:01.432087 20146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:01.449730 20146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:01.461117 20146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:01.461445 20146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:01.463274 20146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:48:01.464640 20146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:48:01.464772 20146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:01.464948 20146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:48:01.465049 20146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:01.480758 20146 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.129:33303
I20260806 02:48:01.480835 20351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.129:33303 every 8 connection(s)
I20260806 02:48:01.482244 20146 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
I20260806 02:48:01.487159 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 20146
I20260806 02:48:01.487335 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance
I20260806 02:48:01.487717 1758 external_mini_cluster.cc:1428] Setting key 56d5c76c9be13ca232cc1ea98dfa645f
Aug 06 02:48:01 dist-test-slave-gmjp krb5kdc[19279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984481, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.183.130@KRBTEST.COM; defaulting to no policy
I20260806 02:48:01.518366 20356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:01.485318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35991 (local address 127.1.183.129:59943)
0806 02:48:01.496652 (+ 11334us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:01.496740 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:01.498083 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:01.499199 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:01.499211 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:01.499675 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:01.500526 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:01.500552 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:01.501889 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:01.501893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:01.503378 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:01.503391 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:01.504329 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:01.504333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:01.504557 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:01.505291 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:01.505314 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:01.507204 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:01.514258 (+ 7054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:01.514268 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:01.514293 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:01.514743 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:01.515151 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:01.515156 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:01.515158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:01.515401 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:01.515985 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:01.515994 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:01.516448 (+ 454us) client_negotiation.cc:769] Sending connection context
0806 02:48:01.516698 (+ 250us) client_negotiation.cc:240] Negotiation successful
0806 02:48:01.517134 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11016,"thread_start_us":214,"threads_started":1}
I20260806 02:48:01.519418 20359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:01.486707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:59943 (local address 127.1.183.190:35991)
0806 02:48:01.495260 (+ 8553us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:01.495266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:01.498516 (+ 3250us) server_negotiation.cc:407] Connection header received
0806 02:48:01.498587 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:01.498590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:01.498666 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:01.498788 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:01.500757 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:01.501678 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:01.503532 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:01.503962 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:01.504053 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:01.507452 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:01.507456 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:01.507459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:01.507492 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:01.514037 (+ 6545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:01.514907 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:01.514911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:01.514913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:01.515023 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:01.515528 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:01.515532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:01.515534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:01.515815 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:01.515919 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:01.519055 (+ 3136us) server_negotiation.cc:299] Negotiation successful
0806 02:48:01.519226 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8436,"thread_start_us":100,"threads_started":1}
I20260806 02:48:01.520691 20352 heartbeater.cc:347] Connected to a master server at 127.1.183.190:35991
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.130@KRBTEST.COM" created.
I20260806 02:48:01.521286 20352 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:01.521931 20352 heartbeater.cc:511] Master 127.1.183.190:35991 requested a full tablet report, sending...
I20260806 02:48:01.524255 20010 ts_manager.cc:194] Registered new tserver with Master: 0eb80f8bbbe64dd7a94680e19017f536 (127.1.183.129:33303)
I20260806 02:48:01.525904 20010 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.129@KRBTEST.COM'} at 127.1.183.129:59943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
Aug 06 02:48:01 dist-test-slave-gmjp krb5kdc[19279](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785984481, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:01.543951 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.183.148:53643/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.130:0
--local_ip_for_outbound_sockets=127.1.183.130
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:35991
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:33657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:48:01.682205 20362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:01.682529 20362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:01.682657 20362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:01.687001 20362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:01.687089 20362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:01.687198 20362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.130
I20260806 02:48:01.693368 20362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:33657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.183.148:53643/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:35991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.130
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:48:01.694552 20362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:01.695739 20362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:01.707652 20378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:01.707652 20371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:01.707877 20372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:01.708716 20362 server_base.cc:1034] running on GCE node
I20260806 02:48:01.709182 20362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:01.710047 20362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:01.711731 20362 hybrid_clock.cc:648] HybridClock initialized: now 1785984481711646 us; error 97 us; skew 500 ppm
Aug 06 02:48:01 dist-test-slave-gmjp krb5kdc[19279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984481, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:01.715436 20362 init.cc:377] Logged in from keytab as kudu/127.1.183.130@KRBTEST.COM (short username kudu)
I20260806 02:48:01.716799 20362 webserver.cc:460] Webserver started at http://127.1.183.130:39407/ using document root <none> and password file <none>
I20260806 02:48:01.717612 20362 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:01.717722 20362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:01.718037 20362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:48:01 dist-test-slave-gmjp krb5kdc[19279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984481, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for HTTP/127.1.183.148@KRBTEST.COM
I20260806 02:48:01.737671 20362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-1/data/instance:
uuid: "f8a8368d2e00447da3b2765ba4be10dd"
format_stamp: "Formatted at 2026-08-06 02:48:01 on dist-test-slave-gmjp"
server_key: "cfa10224b71015e1a321ef55f11fb69c"
server_key_iv: "ffc9b358f78a8a97e63d39b71dc0922b"
server_key_version: "kuduclusterkey@0"
I20260806 02:48:01.738658 20362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance:
uuid: "f8a8368d2e00447da3b2765ba4be10dd"
format_stamp: "Formatted at 2026-08-06 02:48:01 on dist-test-slave-gmjp"
server_key: "cfa10224b71015e1a321ef55f11fb69c"
server_key_iv: "ffc9b358f78a8a97e63d39b71dc0922b"
server_key_version: "kuduclusterkey@0"
I20260806 02:48:01.749643 20362 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.004s sys 0.000s
I20260806 02:48:01.779856 20388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:01.781905 20362 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260806 02:48:01.782094 20362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-1/wal
uuid: "f8a8368d2e00447da3b2765ba4be10dd"
format_stamp: "Formatted at 2026-08-06 02:48:01 on dist-test-slave-gmjp"
server_key: "cfa10224b71015e1a321ef55f11fb69c"
server_key_iv: "ffc9b358f78a8a97e63d39b71dc0922b"
server_key_version: "kuduclusterkey@0"
I20260806 02:48:01.782298 20362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:01.806354 20362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:01.810482 20362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:01.810721 20362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:01.811568 20362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:48:01.812801 20362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:48:01.812885 20362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:01.812958 20362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:48:01.812988 20362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:01.827066 20362 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.130:39199
I20260806 02:48:01.828389 20552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.130:39199 every 8 connection(s)
I20260806 02:48:01.829150 20362 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
I20260806 02:48:01.834661 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 20362
I20260806 02:48:01.834831 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance
I20260806 02:48:01.835168 1758 external_mini_cluster.cc:1428] Setting key e58b280e9d3a3fcb890bc57fdb359cb6
Aug 06 02:48:01 dist-test-slave-gmjp krb5kdc[19279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984481, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.183.131@KRBTEST.COM; defaulting to no policy
I20260806 02:48:01.859009 20359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:01.833532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:47447 (local address 127.1.183.190:35991)
0806 02:48:01.833679 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:01.833684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:01.837020 (+ 3336us) server_negotiation.cc:407] Connection header received
0806 02:48:01.839775 (+ 2755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:01.839779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:01.839846 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:01.839945 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:01.841831 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:01.842658 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:01.844815 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:01.845249 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:01.845359 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:01.849035 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:01.849040 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:01.849043 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:01.849077 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:01.850887 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:01.852308 (+ 1421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:01.852313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:01.852315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:01.852421 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:01.857298 (+ 4877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:01.857304 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:01.857306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:01.857613 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:01.857724 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:01.858642 (+ 918us) server_negotiation.cc:299] Negotiation successful
0806 02:48:01.858808 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:48:01.861150 20555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:01.832091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35991 (local address 127.1.183.130:47447)
0806 02:48:01.836641 (+ 4550us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:01.836725 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:01.837893 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:01.840331 (+ 2438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:01.840343 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:01.840742 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:01.841606 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:01.841640 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:01.842855 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:01.842858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:01.844649 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:01.844666 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:01.845628 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:01.845635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:01.846073 (+ 438us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:01.846848 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:01.846873 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:01.848775 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:01.851124 (+ 2349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:01.851133 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:01.851156 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:01.851561 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:01.852552 (+ 991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:01.852555 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:01.852557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:01.852790 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:01.857822 (+ 5032us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:01.857831 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:01.858273 (+ 442us) client_negotiation.cc:769] Sending connection context
0806 02:48:01.859514 (+ 1241us) client_negotiation.cc:240] Negotiation successful
0806 02:48:01.859952 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4272,"thread_start_us":172,"threads_started":1}
I20260806 02:48:01.862902 20553 heartbeater.cc:347] Connected to a master server at 127.1.183.190:35991
I20260806 02:48:01.863241 20553 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:01.863979 20553 heartbeater.cc:511] Master 127.1.183.190:35991 requested a full tablet report, sending...
I20260806 02:48:01.866123 20010 ts_manager.cc:194] Registered new tserver with Master: f8a8368d2e00447da3b2765ba4be10dd (127.1.183.130:39199)
I20260806 02:48:01.866917 20010 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.130@KRBTEST.COM'} at 127.1.183.130:47447
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
I20260806 02:48:01.887591 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.183.148:53643/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.131:0
--local_ip_for_outbound_sockets=127.1.183.131
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:35991
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:33657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:48:02.061060 20559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:02.061437 20559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:02.061503 20559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:02.067737 20559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:02.067847 20559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:02.067935 20559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.131
I20260806 02:48:02.076856 20559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:33657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.183.148:53643/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:35991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.131
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:48:02.078497 20559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:02.080067 20559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:02.094400 20573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:02.094928 20559 server_base.cc:1034] running on GCE node
W20260806 02:48:02.095546 20576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:02.104434 20569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:02.105195 20559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:02.106240 20559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:02.121896 20559 hybrid_clock.cc:648] HybridClock initialized: now 1785984482121751 us; error 133 us; skew 500 ppm
Aug 06 02:48:02 dist-test-slave-gmjp krb5kdc[19279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984482, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:02.126569 20559 init.cc:377] Logged in from keytab as kudu/127.1.183.131@KRBTEST.COM (short username kudu)
I20260806 02:48:02.127912 20559 webserver.cc:460] Webserver started at http://127.1.183.131:38553/ using document root <none> and password file <none>
I20260806 02:48:02.128769 20559 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:02.128935 20559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:02.129256 20559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:48:02 dist-test-slave-gmjp krb5kdc[19279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984481, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for HTTP/127.1.183.148@KRBTEST.COM
Aug 06 02:48:02 dist-test-slave-gmjp krb5kdc[19279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984482, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for HTTP/127.1.183.148@KRBTEST.COM
I20260806 02:48:02.151643 20559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-2/data/instance:
uuid: "9b2cf1dc561949b39901b169dac420a8"
format_stamp: "Formatted at 2026-08-06 02:48:02 on dist-test-slave-gmjp"
server_key: "a6ad54824284279fa55ad40685e36f28"
server_key_iv: "9a77e46205245aa8245fa798266c6fad"
server_key_version: "kuduclusterkey@0"
I20260806 02:48:02.152546 20559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance:
uuid: "9b2cf1dc561949b39901b169dac420a8"
format_stamp: "Formatted at 2026-08-06 02:48:02 on dist-test-slave-gmjp"
server_key: "a6ad54824284279fa55ad40685e36f28"
server_key_iv: "9a77e46205245aa8245fa798266c6fad"
server_key_version: "kuduclusterkey@0"
I20260806 02:48:02.170733 20559 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.005s sys 0.000s
I20260806 02:48:02.196478 20588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:02.198307 20559 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.002s
I20260806 02:48:02.198486 20559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-2/wal
uuid: "9b2cf1dc561949b39901b169dac420a8"
format_stamp: "Formatted at 2026-08-06 02:48:02 on dist-test-slave-gmjp"
server_key: "a6ad54824284279fa55ad40685e36f28"
server_key_iv: "9a77e46205245aa8245fa798266c6fad"
server_key_version: "kuduclusterkey@0"
I20260806 02:48:02.198633 20559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:02.215929 20559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:02.219897 20559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:02.220223 20559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:02.220994 20559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:48:02.222110 20559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:48:02.222225 20559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:02.222326 20559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:48:02.222389 20559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:02.246078 20559 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.131:36049
I20260806 02:48:02.254424 20761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.131:36049 every 8 connection(s)
I20260806 02:48:02.255024 20559 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
I20260806 02:48:02.265198 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 20559
I20260806 02:48:02.265359 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance
I20260806 02:48:02.265738 1758 external_mini_cluster.cc:1428] Setting key 8c877ea868ae0db58f70fe2cafc94502
Aug 06 02:48:02 dist-test-slave-gmjp krb5kdc[19279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984482, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:48:02.297215 20359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:02.272056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:56021 (local address 127.1.183.190:35991)
0806 02:48:02.272209 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:02.272214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:02.273064 (+ 850us) server_negotiation.cc:407] Connection header received
0806 02:48:02.274369 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:02.274373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:02.274430 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:02.274485 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:02.276293 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:02.277131 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:02.288360 (+ 11229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:02.288829 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:02.288946 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:02.292628 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:02.292633 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:02.292636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:02.292668 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:02.294528 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:02.295244 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:02.295249 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:02.295251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:02.295345 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:02.295779 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:02.295782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:02.295784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:02.296029 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:02.296110 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:02.296883 (+ 773us) server_negotiation.cc:299] Negotiation successful
0806 02:48:02.297037 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 02:48:02.299060 20765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:02.270809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35991 (local address 127.1.183.131:56021)
0806 02:48:02.272893 (+ 2084us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:02.272956 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:02.274092 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:02.274937 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:02.274947 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:02.275350 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:02.276066 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:02.276089 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:02.277310 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:02.277313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:02.278775 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:02.278788 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:02.289305 (+ 10517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:02.289311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:02.289621 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:02.290532 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:02.290558 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:02.292402 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:02.294690 (+ 2288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:02.294698 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:02.294717 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:02.295115 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:02.295452 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:02.295456 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:02.295458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:02.295664 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:02.296209 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:02.296216 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:02.296572 (+ 356us) client_negotiation.cc:769] Sending connection context
0806 02:48:02.297491 (+ 919us) client_negotiation.cc:240] Negotiation successful
0806 02:48:02.297891 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1838,"thread_start_us":199,"threads_started":1}
I20260806 02:48:02.300467 20762 heartbeater.cc:347] Connected to a master server at 127.1.183.190:35991
I20260806 02:48:02.300741 20762 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:02.301545 20762 heartbeater.cc:511] Master 127.1.183.190:35991 requested a full tablet report, sending...
I20260806 02:48:02.302901 20010 ts_manager.cc:194] Registered new tserver with Master: 9b2cf1dc561949b39901b169dac420a8 (127.1.183.131:36049)
I20260806 02:48:02.303671 20010 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.131@KRBTEST.COM'} at 127.1.183.131:56021
I20260806 02:48:02.304116 1758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:48:02.304922 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 20146
I20260806 02:48:02.350713 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 20362
I20260806 02:48:02.360926 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 20559
I20260806 02:48:02.387648 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 19966
I20260806 02:48:02.407837 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:35991
--webserver_interface=127.1.183.190
--webserver_port=38637
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:33657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:35991
--ranger_config_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.183.148:53643/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.183.148:53643/kms with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:48:02.541764 20768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:02.542121 20768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:02.542207 20768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:02.546470 20768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:48:02.546574 20768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:02.546610 20768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:02.546650 20768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:48:02.546687 20768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:48:02.552501 20768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:33657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.183.148:53643/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:35991
--ranger_config_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:35991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=38637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:48:02.553743 20768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:02.554852 20768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:02.563868 20774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:02.563941 20775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:02.564002 20777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:02.564291 20768 server_base.cc:1034] running on GCE node
I20260806 02:48:02.564930 20768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:02.566157 20768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:02.567345 20768 hybrid_clock.cc:648] HybridClock initialized: now 1785984482567311 us; error 52 us; skew 500 ppm
Aug 06 02:48:02 dist-test-slave-gmjp krb5kdc[19279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984482, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:02.570329 20768 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:48:02.571477 20768 webserver.cc:460] Webserver started at http://127.1.183.190:38637/ using document root <none> and password file <none>
I20260806 02:48:02.572077 20768 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:02.572192 20768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 02:48:02 dist-test-slave-gmjp krb5kdc[19279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984482, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for HTTP/127.1.183.148@KRBTEST.COM
I20260806 02:48:02.647835 20768 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:48:02.650435 20796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:02.651775 20768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:48:02.651968 20768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "5275dd1bee554875a224893a748eb006"
format_stamp: "Formatted at 2026-08-06 02:48:00 on dist-test-slave-gmjp"
server_key: "dc27699b094e428dac4e52acbbdf1b4d"
server_key_iv: "0643c6c4ced5cbfcde35fb114742aa8b"
server_key_version: "kuduclusterkey@0"
I20260806 02:48:02.652629 20768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:02.663936 20768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:02.666817 20768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:02.667068 20768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:02.674875 20768 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:35991
I20260806 02:48:02.674897 20880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:35991 every 8 connection(s)
I20260806 02:48:02.676098 20768 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:48:02.678099 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 20768
I20260806 02:48:02.678669 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.183.148:53643/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.129:33303
--local_ip_for_outbound_sockets=127.1.183.129
--tserver_master_addrs=127.1.183.190:35991
--webserver_port=40471
--webserver_interface=127.1.183.129
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:33657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.183.148:53643/kms with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:48:02.681675 20881 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:48:02.683620 20881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006: Bootstrap starting.
I20260806 02:48:02.701249 20881 log.cc:824] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:02.708299 20881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:48:02.708760 20881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006: Bootstrap complete.
I20260806 02:48:02.713188 20881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5275dd1bee554875a224893a748eb006" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35991 } }
I20260806 02:48:02.713629 20881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5275dd1bee554875a224893a748eb006, State: Initialized, Role: FOLLOWER
I20260806 02:48:02.714207 20881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5275dd1bee554875a224893a748eb006" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35991 } }
I20260806 02:48:02.714337 20881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:48:02.714434 20881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:48:02.714555 20881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:48:02.716651 20881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5275dd1bee554875a224893a748eb006" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35991 } }
I20260806 02:48:02.717010 20881 leader_election.cc:302] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5275dd1bee554875a224893a748eb006; no voters:
I20260806 02:48:02.717360 20881 leader_election.cc:288] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 02:48:02.717461 20887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 02:48:02.717666 20887 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [term 2 LEADER]: Becoming Leader. State: Replica: 5275dd1bee554875a224893a748eb006, State: Running, Role: LEADER
I20260806 02:48:02.717976 20887 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5275dd1bee554875a224893a748eb006" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35991 } }
I20260806 02:48:02.718704 20881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:48:02.719426 20888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5275dd1bee554875a224893a748eb006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5275dd1bee554875a224893a748eb006" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35991 } } }
I20260806 02:48:02.719563 20888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [sys.catalog]: This master's current role is: LEADER
I20260806 02:48:02.719872 20889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5275dd1bee554875a224893a748eb006. Latest consensus state: current_term: 2 leader_uuid: "5275dd1bee554875a224893a748eb006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5275dd1bee554875a224893a748eb006" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35991 } } }
I20260806 02:48:02.719965 20889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006 [sys.catalog]: This master's current role is: LEADER
I20260806 02:48:02.720465 20896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:48:02.723798 20896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:48:02.724434 20881 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 02:48:02.724599 20896 catalog_manager.cc:1259] Loaded cluster ID: f28279f310e344958b7a1329941742e7
I20260806 02:48:02.724653 20896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:48:02.727048 20896 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:48:02.727737 20896 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006: Loaded TSK: 0
I20260806 02:48:02.732018 20896 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:48:02.824290 20883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:02.824689 20883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:02.824770 20883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:02.830869 20883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:02.830976 20883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:02.831069 20883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.129
I20260806 02:48:02.839219 20883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:33657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.183.148:53643/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.129:33303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.183.129
--webserver_port=40471
--tserver_master_addrs=127.1.183.190:35991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.129
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:48:02.840730 20883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:02.842103 20883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:02.856030 20935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:02.856344 20883 server_base.cc:1034] running on GCE node
W20260806 02:48:02.856683 20940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:02.856989 20938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:02.857513 20883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:02.858363 20883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:02.859575 20883 hybrid_clock.cc:648] HybridClock initialized: now 1785984482859549 us; error 45 us; skew 500 ppm
Aug 06 02:48:02 dist-test-slave-gmjp krb5kdc[19279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984482, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:02.863474 20883 init.cc:377] Logged in from keytab as kudu/127.1.183.129@KRBTEST.COM (short username kudu)
I20260806 02:48:02.864809 20883 webserver.cc:460] Webserver started at http://127.1.183.129:40471/ using document root <none> and password file <none>
I20260806 02:48:02.865576 20883 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:02.865700 20883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 02:48:02 dist-test-slave-gmjp krb5kdc[19279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984482, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for HTTP/127.1.183.148@KRBTEST.COM
I20260806 02:48:02.903373 20883 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:48:02.907087 20955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:02.908866 20883 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 02:48:02.909060 20883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-0/wal
uuid: "0eb80f8bbbe64dd7a94680e19017f536"
format_stamp: "Formatted at 2026-08-06 02:48:01 on dist-test-slave-gmjp"
server_key: "7cffed46b1cb168818e63483a7d04e75"
server_key_iv: "8b006a8711fe2acda41cd08a25e11765"
server_key_version: "kuduclusterkey@0"
I20260806 02:48:02.909852 20883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:02.942723 20883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:02.946674 20883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:02.946924 20883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:02.947670 20883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:48:02.948776 20883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:48:02.948840 20883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:02.948904 20883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:48:02.948930 20883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:02.961102 20883 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.129:33303
I20260806 02:48:02.962422 20883 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
I20260806 02:48:02.971634 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 20883
I20260806 02:48:02.972298 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.183.148:53643/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.130:39199
--local_ip_for_outbound_sockets=127.1.183.130
--tserver_master_addrs=127.1.183.190:35991
--webserver_port=39407
--webserver_interface=127.1.183.130
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:33657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.183.148:53643/kms with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:48:02.973198 21129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.129:33303 every 8 connection(s)
Aug 06 02:48:02 dist-test-slave-gmjp krb5kdc[19279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984482, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:48:03.007404 21134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:02.965139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35991 (local address 127.1.183.129:50857)
0806 02:48:02.976900 (+ 11761us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:02.976987 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:02.978238 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:02.981800 (+ 3562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:02.981810 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:02.982186 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:02.982914 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:02.982935 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:02.985111 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:02.985114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:02.986499 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:02.986512 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:02.987614 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:02.987620 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:02.987836 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:02.988583 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:02.988605 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:02.990388 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:03.001084 (+ 10696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:03.001094 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:03.001114 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:03.001563 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:03.002131 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:03.002135 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:03.002138 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:03.002358 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:03.005118 (+ 2760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:03.005126 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:03.005499 (+ 373us) client_negotiation.cc:769] Sending connection context
0806 02:48:03.005751 (+ 252us) client_negotiation.cc:240] Negotiation successful
0806 02:48:03.006172 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11438,"thread_start_us":152,"threads_started":1}
I20260806 02:48:03.011018 21133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:02.964943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:50857 (local address 127.1.183.190:35991)
0806 02:48:02.968540 (+ 3597us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:02.968571 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:02.980347 (+ 11776us) server_negotiation.cc:407] Connection header received
0806 02:48:02.980733 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:02.980751 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:02.981240 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:02.983197 (+ 1957us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:02.983548 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:02.984932 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:02.986669 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:02.987392 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:02.988289 (+ 897us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:02.990658 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:02.990666 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:02.990684 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:02.990718 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:03.000755 (+ 10037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:03.001764 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:03.001771 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:03.001777 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:03.001963 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:03.004248 (+ 2285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:03.004253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:03.004255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:03.004913 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:03.005143 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:03.009029 (+ 3886us) server_negotiation.cc:299] Negotiation successful
0806 02:48:03.009914 (+ 885us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3351,"thread_start_us":131,"threads_started":1}
I20260806 02:48:03.014062 21130 heartbeater.cc:347] Connected to a master server at 127.1.183.190:35991
I20260806 02:48:03.014354 21130 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:03.015041 21130 heartbeater.cc:511] Master 127.1.183.190:35991 requested a full tablet report, sending...
W20260806 02:48:03.017612 21130 heartbeater.cc:644] Failed to heartbeat to 127.1.183.190:35991 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 02:48:03.017752 21130 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:03.017975 21130 heartbeater.cc:511] Master 127.1.183.190:35991 requested a full tablet report, sending...
I20260806 02:48:03.018989 21130 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:03.019187 21130 heartbeater.cc:511] Master 127.1.183.190:35991 requested a full tablet report, sending...
W20260806 02:48:03.020033 21130 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:48:03.179201 21136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:03.179638 21136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:03.179741 21136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:03.187616 21136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:03.187801 21136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:03.187963 21136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.130
I20260806 02:48:03.195997 21136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:33657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.183.148:53643/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.130:39199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.183.130
--webserver_port=39407
--tserver_master_addrs=127.1.183.190:35991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.130
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:48:03.197461 21136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:03.198626 21136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:03.209198 21141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:03.209671 21147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:03.209935 21136 server_base.cc:1034] running on GCE node
W20260806 02:48:03.210182 21142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:03.210652 21136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:03.211404 21136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:03.212598 21136 hybrid_clock.cc:648] HybridClock initialized: now 1785984483212553 us; error 51 us; skew 500 ppm
Aug 06 02:48:03 dist-test-slave-gmjp krb5kdc[19279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984483, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:03.216656 21136 init.cc:377] Logged in from keytab as kudu/127.1.183.130@KRBTEST.COM (short username kudu)
I20260806 02:48:03.217883 21136 webserver.cc:460] Webserver started at http://127.1.183.130:39407/ using document root <none> and password file <none>
I20260806 02:48:03.218761 21136 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:03.218942 21136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 02:48:03 dist-test-slave-gmjp krb5kdc[19279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984483, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for HTTP/127.1.183.148@KRBTEST.COM
I20260806 02:48:03.257719 21136 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.004s sys 0.000s
I20260806 02:48:03.261765 21156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:03.263747 21136 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:48:03.263926 21136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-1/wal
uuid: "f8a8368d2e00447da3b2765ba4be10dd"
format_stamp: "Formatted at 2026-08-06 02:48:01 on dist-test-slave-gmjp"
server_key: "cfa10224b71015e1a321ef55f11fb69c"
server_key_iv: "ffc9b358f78a8a97e63d39b71dc0922b"
server_key_version: "kuduclusterkey@0"
I20260806 02:48:03.264825 21136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:03.284379 21136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:03.288852 21136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:03.289121 21136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:03.289987 21136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:48:03.291345 21136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:48:03.291422 21136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:03.291489 21136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:48:03.291529 21136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:03.304544 21136 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.130:39199
I20260806 02:48:03.306360 21136 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
I20260806 02:48:03.311086 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 21136
I20260806 02:48:03.311738 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.183.148:53643/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.131:36049
--local_ip_for_outbound_sockets=127.1.183.131
--tserver_master_addrs=127.1.183.190:35991
--webserver_port=38553
--webserver_interface=127.1.183.131
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:33657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.183.148:53643/kms with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:48:03.322162 21326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.130:39199 every 8 connection(s)
Aug 06 02:48:03 dist-test-slave-gmjp krb5kdc[19279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984483, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:48:03.350296 21133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:03.320412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:35837 (local address 127.1.183.190:35991)
0806 02:48:03.320599 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:03.320606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:03.322857 (+ 2251us) server_negotiation.cc:407] Connection header received
0806 02:48:03.324361 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:03.324366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:03.324439 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:03.324507 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:03.326556 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:03.327404 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:03.329341 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:03.329714 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:03.329805 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:03.345188 (+ 15383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:03.345193 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:03.345196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:03.345237 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:03.347222 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:03.348131 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:03.348138 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:03.348143 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:03.348284 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:03.348760 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:03.348763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:03.348765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:03.349005 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:03.349096 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:03.349940 (+ 844us) server_negotiation.cc:299] Negotiation successful
0806 02:48:03.350115 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 02:48:03.352195 21333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:03.319763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35991 (local address 127.1.183.130:35837)
0806 02:48:03.322626 (+ 2863us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:03.322703 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:03.324004 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:03.325000 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:03.325012 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:03.325604 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:03.326348 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:03.326370 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:03.327586 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:03.327590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:03.329165 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:03.329178 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:03.330051 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:03.330055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:03.330267 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:03.340310 (+ 10043us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:03.340341 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:03.344914 (+ 4573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:03.347455 (+ 2541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:03.347467 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:03.347491 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:03.347973 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:03.348433 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:03.348436 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:03.348438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:03.348647 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:03.349179 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:03.349188 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:03.349599 (+ 411us) client_negotiation.cc:769] Sending connection context
0806 02:48:03.350615 (+ 1016us) client_negotiation.cc:240] Negotiation successful
0806 02:48:03.351035 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2523,"thread_start_us":2262,"threads_started":1}
I20260806 02:48:03.353714 21327 heartbeater.cc:347] Connected to a master server at 127.1.183.190:35991
I20260806 02:48:03.354003 21327 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:03.354593 21327 heartbeater.cc:511] Master 127.1.183.190:35991 requested a full tablet report, sending...
W20260806 02:48:03.356674 21327 heartbeater.cc:644] Failed to heartbeat to 127.1.183.190:35991 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 02:48:03.356834 21327 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:03.357091 21327 heartbeater.cc:511] Master 127.1.183.190:35991 requested a full tablet report, sending...
I20260806 02:48:03.358130 21327 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:03.358332 21327 heartbeater.cc:511] Master 127.1.183.190:35991 requested a full tablet report, sending...
W20260806 02:48:03.359208 21327 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:48:03.514150 21331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:03.514643 21331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:03.514727 21331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:03.524906 21331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:03.525009 21331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:03.525128 21331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.131
I20260806 02:48:03.537890 21331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:33657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.183.148:53643/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.131:36049
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.183.131
--webserver_port=38553
--tserver_master_addrs=127.1.183.190:35991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.131
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:48:03.539568 21331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:03.541195 21331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:03.559886 21346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:03.560273 21342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:03.559890 21341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:03.561525 21331 server_base.cc:1034] running on GCE node
I20260806 02:48:03.562116 21331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:03.563119 21331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:03.564296 21331 hybrid_clock.cc:648] HybridClock initialized: now 1785984483564259 us; error 55 us; skew 500 ppm
Aug 06 02:48:03 dist-test-slave-gmjp krb5kdc[19279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984483, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:03.568583 21331 init.cc:377] Logged in from keytab as kudu/127.1.183.131@KRBTEST.COM (short username kudu)
I20260806 02:48:03.569994 21331 webserver.cc:460] Webserver started at http://127.1.183.131:38553/ using document root <none> and password file <none>
I20260806 02:48:03.570978 21331 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:03.571159 21331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 02:48:03 dist-test-slave-gmjp krb5kdc[19279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984483, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for HTTP/127.1.183.148@KRBTEST.COM
I20260806 02:48:03.606201 21331 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.004s sys 0.000s
I20260806 02:48:03.616518 21363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:03.618660 21331 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.005s sys 0.000s
I20260806 02:48:03.618844 21331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-2/wal
uuid: "9b2cf1dc561949b39901b169dac420a8"
format_stamp: "Formatted at 2026-08-06 02:48:02 on dist-test-slave-gmjp"
server_key: "a6ad54824284279fa55ad40685e36f28"
server_key_iv: "9a77e46205245aa8245fa798266c6fad"
server_key_version: "kuduclusterkey@0"
I20260806 02:48:03.619864 21331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:03.636058 21331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:03.640079 21331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:03.640398 21331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:03.641167 21331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:48:03.642307 21331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:48:03.642395 21331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:03.642470 21331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:48:03.642527 21331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:03.654459 21331 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.131:36049
I20260806 02:48:03.654482 21520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.131:36049 every 8 connection(s)
I20260806 02:48:03.655714 21331 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
I20260806 02:48:03.661516 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 21331
I20260806 02:48:03.671943 21133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:03.662314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45234 (local address 127.1.183.190:35991)
0806 02:48:03.662473 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:03.662479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:03.662784 (+ 305us) server_negotiation.cc:407] Connection header received
0806 02:48:03.662951 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:03.662954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:03.663004 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:03.663088 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:03.664002 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:03.664837 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:03.665853 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:03.666172 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:03.666284 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:03.667945 (+ 1661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:03.667950 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:03.667953 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:03.667985 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:03.669737 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:03.670260 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:03.670263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:03.670265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:03.670364 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:03.670882 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:03.670885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:03.670887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:03.671107 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:03.671299 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:03.671547 (+ 248us) server_negotiation.cc:299] Negotiation successful
0806 02:48:03.671745 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Aug 06 02:48:03 dist-test-slave-gmjp krb5kdc[19279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984483, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:48:03.679289 21528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:03.662568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:45777 (local address 127.1.183.190:35991)
0806 02:48:03.664020 (+ 1452us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:03.664025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:03.664038 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:48:03.664068 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:03.664070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:03.664114 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:03.664228 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:03.667467 (+ 3239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:03.670068 (+ 2601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:03.672010 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:03.672571 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:03.672829 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:03.675559 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:03.675563 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:03.675566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:03.675591 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:03.677149 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:03.677685 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:03.677690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:03.677692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:03.677779 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:03.678153 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:03.678157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:03.678158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:03.678352 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:03.678472 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:03.678990 (+ 518us) server_negotiation.cc:299] Negotiation successful
0806 02:48:03.679120 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":84,"threads_started":1}
I20260806 02:48:03.680727 21526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:03.658547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35991 (local address 127.1.183.131:45777)
0806 02:48:03.659075 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:03.659132 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:03.660110 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:03.664586 (+ 4476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:03.664597 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:03.666057 (+ 1460us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:03.667310 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:03.667336 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:03.670204 (+ 2868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:03.670210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:03.671713 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:03.671729 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:03.672881 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:03.672886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:03.673146 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:03.673865 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:03.673886 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:03.675380 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:03.677284 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:03.677290 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:03.677306 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:03.677574 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:03.677886 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:03.677889 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:03.677891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:03.678049 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:03.678484 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:03.678492 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:03.678750 (+ 258us) client_negotiation.cc:769] Sending connection context
0806 02:48:03.678949 (+ 199us) client_negotiation.cc:240] Negotiation successful
0806 02:48:03.679279 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":140,"threads_started":1}
I20260806 02:48:03.681936 21521 heartbeater.cc:347] Connected to a master server at 127.1.183.190:35991
I20260806 02:48:03.682205 21521 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:03.682735 21521 heartbeater.cc:511] Master 127.1.183.190:35991 requested a full tablet report, sending...
I20260806 02:48:03.684119 20824 ts_manager.cc:194] Registered new tserver with Master: 9b2cf1dc561949b39901b169dac420a8 (127.1.183.131:36049)
I20260806 02:48:03.685750 20824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.131@KRBTEST.COM'} at 127.1.183.131:45777
I20260806 02:48:04.020282 21130 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:04.020602 21130 heartbeater.cc:511] Master 127.1.183.190:35991 requested a full tablet report, sending...
I20260806 02:48:04.021493 20824 ts_manager.cc:194] Registered new tserver with Master: 0eb80f8bbbe64dd7a94680e19017f536 (127.1.183.129:33303)
I20260806 02:48:04.022323 20824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.129@KRBTEST.COM'} at 127.1.183.129:50857
Aug 06 02:48:04 dist-test-slave-gmjp krb5kdc[19279](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785984484, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:04.359503 21327 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:04.360044 21327 heartbeater.cc:511] Master 127.1.183.190:35991 requested a full tablet report, sending...
I20260806 02:48:04.361575 20824 ts_manager.cc:194] Registered new tserver with Master: f8a8368d2e00447da3b2765ba4be10dd (127.1.183.130:39199)
I20260806 02:48:04.362900 20824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.130@KRBTEST.COM'} at 127.1.183.130:35837
I20260806 02:48:04.372263 1758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:48:04.396031 21556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:04.377054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45236 (local address 127.1.183.190:35991)
0806 02:48:04.377844 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:04.377852 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:04.378535 (+ 683us) server_negotiation.cc:407] Connection header received
0806 02:48:04.378995 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:04.378999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:04.379093 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:04.379307 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:04.380674 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:04.382026 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:04.384123 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:04.384751 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:04.384994 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:04.388550 (+ 3556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:04.388556 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:04.388560 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:04.388639 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:04.392409 (+ 3770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:04.393463 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:04.393478 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:04.393490 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:04.393672 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:04.394265 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:04.394279 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:04.394281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:04.394741 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:04.394999 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:04.395425 (+ 426us) server_negotiation.cc:299] Negotiation successful
0806 02:48:04.395719 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":163,"threads_started":1}
I20260806 02:48:04.404511 20824 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:48:04.413599 20824 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:48:04.459254 21563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:04.439538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:36049 (local address 127.0.0.1:34118)
0806 02:48:04.440697 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:04.440802 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:04.441060 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:04.445974 (+ 4914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:04.445994 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:04.446046 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:04.446363 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:04.446384 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:04.448936 (+ 2552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:04.448941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:04.453572 (+ 4631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:04.453585 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:04.458106 (+ 4521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:04.458111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:04.458342 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:04.458447 (+ 105us) client_negotiation.cc:769] Sending connection context
0806 02:48:04.458687 (+ 240us) client_negotiation.cc:240] Negotiation successful
0806 02:48:04.458954 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"thread_start_us":214,"threads_started":1}
I20260806 02:48:04.460556 21564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:04.439775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34118 (local address 127.1.183.131:36049)
0806 02:48:04.441603 (+ 1828us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:04.441624 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:04.442274 (+ 650us) server_negotiation.cc:407] Connection header received
0806 02:48:04.442450 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:04.442460 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:04.442914 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:04.443215 (+ 301us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:04.446836 (+ 3621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:04.448685 (+ 1849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:04.453902 (+ 5217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:04.457343 (+ 3441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:04.457564 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:04.457978 (+ 414us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:04.459963 (+ 1985us) server_negotiation.cc:299] Negotiation successful
0806 02:48:04.460279 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1640,"thread_start_us":150,"threads_started":1}
I20260806 02:48:04.468479 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:04.445065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:39199 (local address 127.0.0.1:60764)
0806 02:48:04.452660 (+ 7595us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:04.452690 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:04.452861 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:04.456647 (+ 3786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:04.456650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:04.456695 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:04.456951 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:04.456971 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:04.459238 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:04.459241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:04.464732 (+ 5491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:04.464757 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:04.467469 (+ 2712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:04.467475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:04.467680 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:04.467756 (+ 76us) client_negotiation.cc:769] Sending connection context
0806 02:48:04.467906 (+ 150us) client_negotiation.cc:240] Negotiation successful
0806 02:48:04.468080 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7401,"thread_start_us":195,"threads_started":1}
I20260806 02:48:04.467993 21404 tablet_service.cc:1467] Processing CreateTablet for tablet e9c75876b1b141aa8cc8c5d6f44a9dc9 (DEFAULT_TABLE table=test-table [id=a65de29611694bfbbb92b5ebc96b04b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:48:04.471915 21567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:04.444575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60764 (local address 127.1.183.130:39199)
0806 02:48:04.451833 (+ 7258us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:04.451843 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:04.453077 (+ 1234us) server_negotiation.cc:407] Connection header received
0806 02:48:04.453242 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:04.453279 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:04.453711 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:04.454091 (+ 380us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:04.457225 (+ 3134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:04.458993 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:04.465086 (+ 6093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:04.466762 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:04.466991 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:04.467347 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:04.469471 (+ 2124us) server_negotiation.cc:299] Negotiation successful
0806 02:48:04.471623 (+ 2152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7040,"thread_start_us":133,"threads_started":1}
I20260806 02:48:04.472976 21404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9c75876b1b141aa8cc8c5d6f44a9dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:48:04.478644 21224 tablet_service.cc:1467] Processing CreateTablet for tablet e9c75876b1b141aa8cc8c5d6f44a9dc9 (DEFAULT_TABLE table=test-table [id=a65de29611694bfbbb92b5ebc96b04b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:48:04.481196 21224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9c75876b1b141aa8cc8c5d6f44a9dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:48:04.495922 21565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:04.441676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:33303 (local address 127.0.0.1:40430)
0806 02:48:04.449318 (+ 7642us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:04.449338 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:04.449518 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:04.451375 (+ 1857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:04.451379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:04.451412 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:04.452095 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:04.452106 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:04.469375 (+ 17269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:04.469389 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:04.473479 (+ 4090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:04.473494 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:04.494930 (+ 21436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:04.494935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:04.495170 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:04.495226 (+ 56us) client_negotiation.cc:769] Sending connection context
0806 02:48:04.495414 (+ 188us) client_negotiation.cc:240] Negotiation successful
0806 02:48:04.495597 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7345,"mutex_wait_us":406,"thread_start_us":238,"threads_started":1}
I20260806 02:48:04.496809 21570 tablet_bootstrap.cc:492] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 9b2cf1dc561949b39901b169dac420a8: Bootstrap starting.
I20260806 02:48:04.498919 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:04.442556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40430 (local address 127.1.183.129:33303)
0806 02:48:04.450074 (+ 7518us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:04.450092 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:04.450146 (+ 54us) server_negotiation.cc:407] Connection header received
0806 02:48:04.450332 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:04.450355 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:04.450818 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:04.451179 (+ 361us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:04.454640 (+ 3461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:04.456426 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:04.492414 (+ 35988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:04.494119 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:04.494334 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:04.494681 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:04.498400 (+ 3719us) server_negotiation.cc:299] Negotiation successful
0806 02:48:04.498655 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7283,"thread_start_us":235,"threads_started":1}
I20260806 02:48:04.501890 21570 tablet_bootstrap.cc:654] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 9b2cf1dc561949b39901b169dac420a8: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:04.503978 21570 log.cc:824] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 9b2cf1dc561949b39901b169dac420a8: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:04.506091 20999 tablet_service.cc:1467] Processing CreateTablet for tablet e9c75876b1b141aa8cc8c5d6f44a9dc9 (DEFAULT_TABLE table=test-table [id=a65de29611694bfbbb92b5ebc96b04b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:48:04.508688 20999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9c75876b1b141aa8cc8c5d6f44a9dc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:48:04.519003 21570 tablet_bootstrap.cc:492] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 9b2cf1dc561949b39901b169dac420a8: No bootstrap required, opened a new log
I20260806 02:48:04.519474 21570 ts_tablet_manager.cc:1397] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 9b2cf1dc561949b39901b169dac420a8: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.000s
I20260806 02:48:04.524607 21572 tablet_bootstrap.cc:492] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P f8a8368d2e00447da3b2765ba4be10dd: Bootstrap starting.
I20260806 02:48:04.529904 21572 tablet_bootstrap.cc:654] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P f8a8368d2e00447da3b2765ba4be10dd: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:04.530570 21570 raft_consensus.cc:348] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 9b2cf1dc561949b39901b169dac420a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b2cf1dc561949b39901b169dac420a8" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 36049 } } peers { permanent_uuid: "0eb80f8bbbe64dd7a94680e19017f536" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 33303 } } peers { permanent_uuid: "f8a8368d2e00447da3b2765ba4be10dd" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39199 } }
I20260806 02:48:04.531231 21570 raft_consensus.cc:374] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 9b2cf1dc561949b39901b169dac420a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:04.531440 21570 raft_consensus.cc:729] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 9b2cf1dc561949b39901b169dac420a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b2cf1dc561949b39901b169dac420a8, State: Initialized, Role: FOLLOWER
I20260806 02:48:04.532514 21570 consensus_queue.cc:260] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 9b2cf1dc561949b39901b169dac420a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b2cf1dc561949b39901b169dac420a8" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 36049 } } peers { permanent_uuid: "0eb80f8bbbe64dd7a94680e19017f536" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 33303 } } peers { permanent_uuid: "f8a8368d2e00447da3b2765ba4be10dd" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39199 } }
I20260806 02:48:04.534843 21570 ts_tablet_manager.cc:1428] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 9b2cf1dc561949b39901b169dac420a8: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260806 02:48:04.535634 21521 heartbeater.cc:503] Master 127.1.183.190:35991 was elected leader, sending a full tablet report...
I20260806 02:48:04.539274 21572 log.cc:824] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P f8a8368d2e00447da3b2765ba4be10dd: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:04.540787 21574 tablet_bootstrap.cc:492] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 0eb80f8bbbe64dd7a94680e19017f536: Bootstrap starting.
I20260806 02:48:04.544519 21574 tablet_bootstrap.cc:654] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 0eb80f8bbbe64dd7a94680e19017f536: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:04.547178 21572 tablet_bootstrap.cc:492] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P f8a8368d2e00447da3b2765ba4be10dd: No bootstrap required, opened a new log
I20260806 02:48:04.547609 21572 ts_tablet_manager.cc:1397] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P f8a8368d2e00447da3b2765ba4be10dd: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.000s
I20260806 02:48:04.549541 21574 log.cc:824] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 0eb80f8bbbe64dd7a94680e19017f536: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:04.552737 21574 tablet_bootstrap.cc:492] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 0eb80f8bbbe64dd7a94680e19017f536: No bootstrap required, opened a new log
I20260806 02:48:04.553014 21574 ts_tablet_manager.cc:1397] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 0eb80f8bbbe64dd7a94680e19017f536: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.003s
I20260806 02:48:04.554271 21572 raft_consensus.cc:348] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P f8a8368d2e00447da3b2765ba4be10dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b2cf1dc561949b39901b169dac420a8" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 36049 } } peers { permanent_uuid: "0eb80f8bbbe64dd7a94680e19017f536" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 33303 } } peers { permanent_uuid: "f8a8368d2e00447da3b2765ba4be10dd" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39199 } }
I20260806 02:48:04.554756 21572 raft_consensus.cc:374] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P f8a8368d2e00447da3b2765ba4be10dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:04.554826 21572 raft_consensus.cc:729] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P f8a8368d2e00447da3b2765ba4be10dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8a8368d2e00447da3b2765ba4be10dd, State: Initialized, Role: FOLLOWER
I20260806 02:48:04.555761 21572 consensus_queue.cc:260] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P f8a8368d2e00447da3b2765ba4be10dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b2cf1dc561949b39901b169dac420a8" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 36049 } } peers { permanent_uuid: "0eb80f8bbbe64dd7a94680e19017f536" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 33303 } } peers { permanent_uuid: "f8a8368d2e00447da3b2765ba4be10dd" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39199 } }
I20260806 02:48:04.557801 21572 ts_tablet_manager.cc:1428] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P f8a8368d2e00447da3b2765ba4be10dd: Time spent starting tablet: real 0.010s user 0.005s sys 0.003s
I20260806 02:48:04.558410 21327 heartbeater.cc:503] Master 127.1.183.190:35991 was elected leader, sending a full tablet report...
I20260806 02:48:04.558727 21574 raft_consensus.cc:348] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 0eb80f8bbbe64dd7a94680e19017f536 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b2cf1dc561949b39901b169dac420a8" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 36049 } } peers { permanent_uuid: "0eb80f8bbbe64dd7a94680e19017f536" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 33303 } } peers { permanent_uuid: "f8a8368d2e00447da3b2765ba4be10dd" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39199 } }
I20260806 02:48:04.559175 21574 raft_consensus.cc:374] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 0eb80f8bbbe64dd7a94680e19017f536 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:04.559242 21574 raft_consensus.cc:729] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 0eb80f8bbbe64dd7a94680e19017f536 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eb80f8bbbe64dd7a94680e19017f536, State: Initialized, Role: FOLLOWER
I20260806 02:48:04.559862 21574 consensus_queue.cc:260] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 0eb80f8bbbe64dd7a94680e19017f536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b2cf1dc561949b39901b169dac420a8" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 36049 } } peers { permanent_uuid: "0eb80f8bbbe64dd7a94680e19017f536" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 33303 } } peers { permanent_uuid: "f8a8368d2e00447da3b2765ba4be10dd" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39199 } }
I20260806 02:48:04.561244 21574 ts_tablet_manager.cc:1428] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 0eb80f8bbbe64dd7a94680e19017f536: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260806 02:48:04.561717 21130 heartbeater.cc:503] Master 127.1.183.190:35991 was elected leader, sending a full tablet report...
W20260806 02:48:04.574654 21329 tablet.cc:2369] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P f8a8368d2e00447da3b2765ba4be10dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:48:04.659195 21525 tablet.cc:2369] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 9b2cf1dc561949b39901b169dac420a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 02:48:04 dist-test-slave-gmjp krb5kdc[19279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984484, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for HTTP/127.1.183.148@KRBTEST.COM
W20260806 02:48:04.734817 21131 tablet.cc:2369] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 0eb80f8bbbe64dd7a94680e19017f536: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:48:04.864504 21578 raft_consensus.cc:482] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 0eb80f8bbbe64dd7a94680e19017f536 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:48:04.865464 21578 raft_consensus.cc:504] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 0eb80f8bbbe64dd7a94680e19017f536 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b2cf1dc561949b39901b169dac420a8" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 36049 } } peers { permanent_uuid: "0eb80f8bbbe64dd7a94680e19017f536" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 33303 } } peers { permanent_uuid: "f8a8368d2e00447da3b2765ba4be10dd" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39199 } }
I20260806 02:48:04.873472 21578 leader_election.cc:288] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 0eb80f8bbbe64dd7a94680e19017f536 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b2cf1dc561949b39901b169dac420a8 (127.1.183.131:36049), f8a8368d2e00447da3b2765ba4be10dd (127.1.183.130:39199)
I20260806 02:48:04.902688 21579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:04.874644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:36049 (local address 127.1.183.129:58041)
0806 02:48:04.877791 (+ 3147us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:04.877836 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:04.878089 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:04.878717 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:04.878720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:04.878737 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:04.879670 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:04.879704 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:04.886327 (+ 6623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:04.886362 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:04.890402 (+ 4040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:04.890452 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:04.898180 (+ 7728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:04.898203 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:04.898993 (+ 790us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:04.899283 (+ 290us) client_negotiation.cc:769] Sending connection context
0806 02:48:04.900024 (+ 741us) client_negotiation.cc:240] Negotiation successful
0806 02:48:04.900894 (+ 870us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2907,"thread_start_us":1324,"threads_started":1}
I20260806 02:48:04.902770 21564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:04.877735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:58041 (local address 127.1.183.131:36049)
0806 02:48:04.877983 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:04.877988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:04.878036 (+ 48us) server_negotiation.cc:407] Connection header received
0806 02:48:04.878356 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:04.878376 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:04.878477 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:04.878695 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:04.880496 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:04.885581 (+ 5085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:04.891270 (+ 5689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:04.897508 (+ 6238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:04.898137 (+ 629us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:04.898549 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:04.900766 (+ 2217us) server_negotiation.cc:299] Negotiation successful
0806 02:48:04.901657 (+ 891us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260806 02:48:04.909165 21438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9c75876b1b141aa8cc8c5d6f44a9dc9" candidate_uuid: "0eb80f8bbbe64dd7a94680e19017f536" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b2cf1dc561949b39901b169dac420a8" is_pre_election: true
I20260806 02:48:04.912745 21581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:04.875218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:39199 (local address 127.1.183.129:40467)
0806 02:48:04.878338 (+ 3120us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:04.878355 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:04.878615 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:04.881377 (+ 2762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:04.881471 (+ 94us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:04.881704 (+ 233us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:04.883564 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:04.883667 (+ 103us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:04.891866 (+ 8199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:04.891936 (+ 70us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:04.899233 (+ 7297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:04.899288 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:04.907569 (+ 8281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:04.907593 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:04.908816 (+ 1223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:04.909165 (+ 349us) client_negotiation.cc:769] Sending connection context
0806 02:48:04.910113 (+ 948us) client_negotiation.cc:240] Negotiation successful
0806 02:48:04.910944 (+ 831us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2979,"thread_start_us":1067,"threads_started":1}
I20260806 02:48:04.912485 21438 raft_consensus.cc:2454] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 9b2cf1dc561949b39901b169dac420a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0eb80f8bbbe64dd7a94680e19017f536 in term 0.
I20260806 02:48:04.916631 21567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:04.875969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:40467 (local address 127.1.183.130:39199)
0806 02:48:04.877038 (+ 1069us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:04.877094 (+ 56us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:04.878587 (+ 1493us) server_negotiation.cc:407] Connection header received
0806 02:48:04.879179 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:04.879258 (+ 79us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:04.879857 (+ 599us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:04.880138 (+ 281us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:04.884624 (+ 4486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:04.890046 (+ 5422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:04.900330 (+ 10284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:04.906375 (+ 6045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:04.908936 (+ 2561us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:04.909505 (+ 569us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:04.914282 (+ 4777us) server_negotiation.cc:299] Negotiation successful
0806 02:48:04.915254 (+ 972us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406}
I20260806 02:48:04.919281 20958 leader_election.cc:302] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 0eb80f8bbbe64dd7a94680e19017f536 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eb80f8bbbe64dd7a94680e19017f536, 9b2cf1dc561949b39901b169dac420a8; no voters:
I20260806 02:48:04.922008 21577 raft_consensus.cc:482] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P f8a8368d2e00447da3b2765ba4be10dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:48:04.922331 21578 raft_consensus.cc:2781] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 0eb80f8bbbe64dd7a94680e19017f536 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:48:04.923115 21578 raft_consensus.cc:482] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 0eb80f8bbbe64dd7a94680e19017f536 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:48:04.923619 21578 raft_consensus.cc:3037] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 0eb80f8bbbe64dd7a94680e19017f536 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:04.923781 21577 raft_consensus.cc:504] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P f8a8368d2e00447da3b2765ba4be10dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b2cf1dc561949b39901b169dac420a8" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 36049 } } peers { permanent_uuid: "0eb80f8bbbe64dd7a94680e19017f536" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 33303 } } peers { permanent_uuid: "f8a8368d2e00447da3b2765ba4be10dd" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39199 } }
I20260806 02:48:04.921388 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9c75876b1b141aa8cc8c5d6f44a9dc9" candidate_uuid: "0eb80f8bbbe64dd7a94680e19017f536" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8a8368d2e00447da3b2765ba4be10dd" is_pre_election: true
I20260806 02:48:04.929375 21233 raft_consensus.cc:2454] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P f8a8368d2e00447da3b2765ba4be10dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0eb80f8bbbe64dd7a94680e19017f536 in term 0.
I20260806 02:48:04.935822 21577 leader_election.cc:288] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P f8a8368d2e00447da3b2765ba4be10dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b2cf1dc561949b39901b169dac420a8 (127.1.183.131:36049), 0eb80f8bbbe64dd7a94680e19017f536 (127.1.183.129:33303)
I20260806 02:48:04.945829 21578 raft_consensus.cc:504] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 0eb80f8bbbe64dd7a94680e19017f536 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b2cf1dc561949b39901b169dac420a8" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 36049 } } peers { permanent_uuid: "0eb80f8bbbe64dd7a94680e19017f536" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 33303 } } peers { permanent_uuid: "f8a8368d2e00447da3b2765ba4be10dd" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39199 } }
I20260806 02:48:04.954370 21438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9c75876b1b141aa8cc8c5d6f44a9dc9" candidate_uuid: "0eb80f8bbbe64dd7a94680e19017f536" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b2cf1dc561949b39901b169dac420a8"
I20260806 02:48:04.955941 21438 raft_consensus.cc:3037] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 9b2cf1dc561949b39901b169dac420a8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:04.967180 21438 raft_consensus.cc:2454] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 9b2cf1dc561949b39901b169dac420a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0eb80f8bbbe64dd7a94680e19017f536 in term 1.
I20260806 02:48:04.970003 20958 leader_election.cc:302] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 0eb80f8bbbe64dd7a94680e19017f536 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eb80f8bbbe64dd7a94680e19017f536, 9b2cf1dc561949b39901b169dac420a8; no voters:
I20260806 02:48:04.973846 21586 raft_consensus.cc:2781] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 0eb80f8bbbe64dd7a94680e19017f536 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:48:04.973570 21233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9c75876b1b141aa8cc8c5d6f44a9dc9" candidate_uuid: "0eb80f8bbbe64dd7a94680e19017f536" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8a8368d2e00447da3b2765ba4be10dd"
I20260806 02:48:04.974645 21233 raft_consensus.cc:3037] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P f8a8368d2e00447da3b2765ba4be10dd [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:04.975704 21586 raft_consensus.cc:686] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 0eb80f8bbbe64dd7a94680e19017f536 [term 1 LEADER]: Becoming Leader. State: Replica: 0eb80f8bbbe64dd7a94680e19017f536, State: Running, Role: LEADER
I20260806 02:48:04.978709 21586 consensus_queue.cc:237] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 0eb80f8bbbe64dd7a94680e19017f536 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b2cf1dc561949b39901b169dac420a8" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 36049 } } peers { permanent_uuid: "0eb80f8bbbe64dd7a94680e19017f536" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 33303 } } peers { permanent_uuid: "f8a8368d2e00447da3b2765ba4be10dd" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39199 } }
I20260806 02:48:04.988116 21564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:04.940250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:37919 (local address 127.1.183.131:36049)
0806 02:48:04.941266 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:04.941296 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:04.943044 (+ 1748us) server_negotiation.cc:407] Connection header received
0806 02:48:04.944597 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:04.944628 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:04.945035 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:04.945516 (+ 481us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:04.952998 (+ 7482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:04.960315 (+ 7317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:04.971598 (+ 11283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:04.978070 (+ 6472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:04.979494 (+ 1424us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:04.980756 (+ 1262us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:04.984637 (+ 3881us) server_negotiation.cc:299] Negotiation successful
0806 02:48:04.985221 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368}
I20260806 02:48:04.988659 21578 leader_election.cc:288] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 0eb80f8bbbe64dd7a94680e19017f536 [CANDIDATE]: Term 1 election: Requested vote from peers 9b2cf1dc561949b39901b169dac420a8 (127.1.183.131:36049), f8a8368d2e00447da3b2765ba4be10dd (127.1.183.130:39199)
I20260806 02:48:04.996331 21584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:04.938426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:36049 (local address 127.1.183.130:37919)
0806 02:48:04.942222 (+ 3796us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:04.942389 (+ 167us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:04.943658 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:04.947894 (+ 4236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:04.947956 (+ 62us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:04.948128 (+ 172us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:04.950024 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:04.950075 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:04.961549 (+ 11474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:04.961579 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:04.969866 (+ 8287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:04.969932 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:04.981739 (+ 11807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:04.981745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:04.983001 (+ 1256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:04.983225 (+ 224us) client_negotiation.cc:769] Sending connection context
0806 02:48:04.989537 (+ 6312us) client_negotiation.cc:240] Negotiation successful
0806 02:48:04.990415 (+ 878us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3339,"mutex_wait_us":5,"spinlock_wait_cycles":177152,"thread_start_us":888,"threads_started":1}
I20260806 02:48:05.000411 21233 raft_consensus.cc:2454] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P f8a8368d2e00447da3b2765ba4be10dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0eb80f8bbbe64dd7a94680e19017f536 in term 1.
I20260806 02:48:05.000382 21438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9c75876b1b141aa8cc8c5d6f44a9dc9" candidate_uuid: "f8a8368d2e00447da3b2765ba4be10dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b2cf1dc561949b39901b169dac420a8" is_pre_election: true
I20260806 02:48:05.000882 21438 raft_consensus.cc:2379] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 9b2cf1dc561949b39901b169dac420a8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8a8368d2e00447da3b2765ba4be10dd in current term 1: Already voted for candidate 0eb80f8bbbe64dd7a94680e19017f536 in this term.
I20260806 02:48:05.001417 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:04.938951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:34899 (local address 127.1.183.129:33303)
0806 02:48:04.939799 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:04.939861 (+ 62us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:04.943046 (+ 3185us) server_negotiation.cc:407] Connection header received
0806 02:48:04.944869 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:04.944914 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:04.945469 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:04.945935 (+ 466us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:04.953664 (+ 7729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:04.957961 (+ 4297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:04.991094 (+ 33133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:04.995183 (+ 4089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:04.996283 (+ 1100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:04.998644 (+ 2361us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:05.000981 (+ 2337us) server_negotiation.cc:299] Negotiation successful
0806 02:48:05.001183 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212}
I20260806 02:48:05.013958 21585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:04.938037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:33303 (local address 127.1.183.130:34899)
0806 02:48:04.942017 (+ 3980us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:04.942145 (+ 128us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:04.943658 (+ 1513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:04.949542 (+ 5884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:04.949624 (+ 82us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:04.949677 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:04.951982 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:04.952103 (+ 121us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:04.959450 (+ 7347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:04.959508 (+ 58us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:04.968678 (+ 9170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:04.968767 (+ 89us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:05.000246 (+ 31479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:05.000251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:05.000662 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:05.000695 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 02:48:05.001993 (+ 1298us) client_negotiation.cc:240] Negotiation successful
0806 02:48:05.002162 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3412,"mutex_wait_us":1796,"spinlock_wait_cycles":60672,"thread_start_us":2025,"threads_started":1}
I20260806 02:48:05.025794 21052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9c75876b1b141aa8cc8c5d6f44a9dc9" candidate_uuid: "f8a8368d2e00447da3b2765ba4be10dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eb80f8bbbe64dd7a94680e19017f536" is_pre_election: true
I20260806 02:48:05.029989 20824 catalog_manager.cc:5495] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 0eb80f8bbbe64dd7a94680e19017f536 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0eb80f8bbbe64dd7a94680e19017f536 (127.1.183.129). New cstate: current_term: 1 leader_uuid: "0eb80f8bbbe64dd7a94680e19017f536" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b2cf1dc561949b39901b169dac420a8" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 36049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0eb80f8bbbe64dd7a94680e19017f536" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 33303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8a8368d2e00447da3b2765ba4be10dd" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39199 } health_report { overall_health: UNKNOWN } } }
I20260806 02:48:05.036286 21159 leader_election.cc:302] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P f8a8368d2e00447da3b2765ba4be10dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8a8368d2e00447da3b2765ba4be10dd; no voters: 0eb80f8bbbe64dd7a94680e19017f536, 9b2cf1dc561949b39901b169dac420a8
I20260806 02:48:05.040428 21577 raft_consensus.cc:2726] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P f8a8368d2e00447da3b2765ba4be10dd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 02:48:05.193679 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:05.185299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40438 (local address 127.1.183.129:33303)
0806 02:48:05.185708 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:05.185714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:05.186484 (+ 770us) server_negotiation.cc:407] Connection header received
0806 02:48:05.186929 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:05.186933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:05.187071 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:05.187298 (+ 227us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:48:05.187775 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:05.188837 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:05.190749 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:05.191368 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:05.191457 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:05.192075 (+ 618us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:48:05.193115 (+ 1040us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:48:05.193227 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:05.193455 (+ 228us) server_negotiation.cc:299] Negotiation successful
0806 02:48:05.193525 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 02:48:05.204731 21233 raft_consensus.cc:1261] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P f8a8368d2e00447da3b2765ba4be10dd [term 1 FOLLOWER]: Refusing update from remote peer 0eb80f8bbbe64dd7a94680e19017f536: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:48:05.204710 21438 raft_consensus.cc:1261] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 9b2cf1dc561949b39901b169dac420a8 [term 1 FOLLOWER]: Refusing update from remote peer 0eb80f8bbbe64dd7a94680e19017f536: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:48:05.205895 21586 consensus_queue.cc:1035] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 0eb80f8bbbe64dd7a94680e19017f536 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b2cf1dc561949b39901b169dac420a8" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 36049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:48:05.206223 21587 consensus_queue.cc:1035] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 0eb80f8bbbe64dd7a94680e19017f536 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8a8368d2e00447da3b2765ba4be10dd" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 39199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:48:05.216768 21598 mvcc.cc:205] Tried to move back new op lower bound from 7315392451388641280 to 7315392450506727424. Current Snapshot: MvccSnapshot[applied={T|T < 7315392451388641280}]
I20260806 02:48:05.225354 21600 mvcc.cc:205] Tried to move back new op lower bound from 7315392451388641280 to 7315392450506727424. Current Snapshot: MvccSnapshot[applied={T|T < 7315392451388641280}]
I20260806 02:48:05.230836 21601 mvcc.cc:205] Tried to move back new op lower bound from 7315392451388641280 to 7315392450506727424. Current Snapshot: MvccSnapshot[applied={T|T < 7315392451388641280}]
I20260806 02:48:05.239022 20823 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:48:05.239497 20823 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:48:05.243654 20823 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5275dd1bee554875a224893a748eb006: Sending DeleteTablet for 3 replicas of tablet e9c75876b1b141aa8cc8c5d6f44a9dc9
I20260806 02:48:05.244766 21404 tablet_service.cc:1514] Processing DeleteTablet for tablet e9c75876b1b141aa8cc8c5d6f44a9dc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:48:05 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:34118
I20260806 02:48:05.245962 21224 tablet_service.cc:1514] Processing DeleteTablet for tablet e9c75876b1b141aa8cc8c5d6f44a9dc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:48:05 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:60764
I20260806 02:48:05.246268 20999 tablet_service.cc:1514] Processing DeleteTablet for tablet e9c75876b1b141aa8cc8c5d6f44a9dc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:48:05 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:40430
I20260806 02:48:05.248576 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 20883
I20260806 02:48:05.259308 21612 tablet_replica.cc:330] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 9b2cf1dc561949b39901b169dac420a8: stopping tablet replica
I20260806 02:48:05.260425 21612 raft_consensus.cc:2229] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 9b2cf1dc561949b39901b169dac420a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:48:05.261384 21612 raft_consensus.cc:2258] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 9b2cf1dc561949b39901b169dac420a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:48:05.267725 21612 ts_tablet_manager.cc:1905] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 9b2cf1dc561949b39901b169dac420a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:48:05.271310 21615 tablet_replica.cc:330] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P f8a8368d2e00447da3b2765ba4be10dd: stopping tablet replica
I20260806 02:48:05.272234 21615 raft_consensus.cc:2229] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P f8a8368d2e00447da3b2765ba4be10dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:48:05.272969 21615 raft_consensus.cc:2258] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P f8a8368d2e00447da3b2765ba4be10dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:48:05.275759 21615 ts_tablet_manager.cc:1905] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P f8a8368d2e00447da3b2765ba4be10dd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 02:48:05.280584 20801 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.183.129:33303 (error 108)
I20260806 02:48:05.280998 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 21136
I20260806 02:48:05.283572 21612 ts_tablet_manager.cc:1918] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 9b2cf1dc561949b39901b169dac420a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:48:05.286849 21612 log.cc:1196] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 9b2cf1dc561949b39901b169dac420a8: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984313887056-1758-0/minicluster-data/ts-2/wal/wals/e9c75876b1b141aa8cc8c5d6f44a9dc9
I20260806 02:48:05.288045 21612 ts_tablet_manager.cc:1939] T e9c75876b1b141aa8cc8c5d6f44a9dc9 P 9b2cf1dc561949b39901b169dac420a8: Deleting consensus metadata
I20260806 02:48:05.290906 20799 catalog_manager.cc:4841] TS 9b2cf1dc561949b39901b169dac420a8 (127.1.183.131:36049): tablet e9c75876b1b141aa8cc8c5d6f44a9dc9 (table test-table [id=a65de29611694bfbbb92b5ebc96b04b4]) successfully deleted
I20260806 02:48:05.303527 21621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:05.296248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:33303 (local address 127.0.0.1:40440)
0806 02:48:05.302893 (+ 6645us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:05.303154 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.129:33303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6446,"thread_start_us":163,"threads_started":1}
W20260806 02:48:05.304345 20801 catalog_manager.cc:4584] TS 0eb80f8bbbe64dd7a94680e19017f536 (127.1.183.129:33303): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e9c75876b1b141aa8cc8c5d6f44a9dc9: Network error: Client connection negotiation failed: client connection to 127.1.183.129:33303: connect: Connection refused (error 111)
W20260806 02:48:05.318639 21621 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:48:05.317223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:39199 (local address 127.0.0.1:60780)
0806 02:48:05.317613 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:05.317632 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:05.318076 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:05.318341 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.130:39199: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":246}
I20260806 02:48:05.318588 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 21331
I20260806 02:48:05.338047 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 20768
I20260806 02:48:05.360736 1758 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 02:48:05.721524 1758 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 02:48:05.721771 1758 mini_ranger.cc:66] Stopping Ranger...
I20260806 02:48:06.100431 1758 mini_ranger.cc:68] Stopped Ranger
2026-08-06 02:48:06.100 UTC [19325] LOG: received smart shutdown request
2026-08-06 02:48:06.107 UTC [19325] LOG: background worker "logical replication launcher" (PID 19343) exited with exit code 1
2026-08-06 02:48:06.108 UTC [19332] LOG: shutting down
2026-08-06 02:48:06.310 UTC [19325] LOG: database system is shut down
2026-08-06T02:48:06Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52011 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:48:06 dist-test-slave-gmjp krb5kdc[21635](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:48:06 dist-test-slave-gmjp krb5kdc[21635](info): set up 2 sockets
Aug 06 02:48:06 dist-test-slave-gmjp krb5kdc[21635](info): commencing operation
krb5kdc: starting...
W20260806 02:48:07.684221 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.221s user 0.002s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:48:07 dist-test-slave-gmjp krb5kdc[21635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:48:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:48:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
I20260806 02:48:07.818936 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:38311
--webserver_interface=127.1.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:41965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:38311,127.1.183.189:44967,127.1.183.188:38519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:48:07.940071 21654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:07.940478 21654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:07.940569 21654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:07.944808 21654 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:48:07.944926 21654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:07.944963 21654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:07.945005 21654 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:48:07.945044 21654 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:48:07.950842 21654 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:41965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:38311,127.1.183.189:44967,127.1.183.188:38519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:38311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:48:07.952172 21654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:07.953332 21654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:07.962944 21665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:07.962955 21666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:07.962944 21668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:07.963397 21654 server_base.cc:1034] running on GCE node
I20260806 02:48:07.964040 21654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:07.965250 21654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:07.966425 21654 hybrid_clock.cc:648] HybridClock initialized: now 1785984487966405 us; error 34 us; skew 500 ppm
Aug 06 02:48:07 dist-test-slave-gmjp krb5kdc[21635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984487, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:07.969441 21654 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:48:07.970418 21654 webserver.cc:460] Webserver started at http://127.1.183.190:33531/ using document root <none> and password file <none>
I20260806 02:48:07.970965 21654 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:07.971026 21654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:07.971215 21654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:48:07.973397 21654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "98cf6ce0391f4f529d8391a5dd9e4faf"
format_stamp: "Formatted at 2026-08-06 02:48:07 on dist-test-slave-gmjp"
server_key: "a15950433a0d62b8873d6c487c4cbd0c"
server_key_iv: "accfff2d42dc682fd97e8e94ac07b622"
server_key_version: "encryptionkey@0"
I20260806 02:48:07.973939 21654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "98cf6ce0391f4f529d8391a5dd9e4faf"
format_stamp: "Formatted at 2026-08-06 02:48:07 on dist-test-slave-gmjp"
server_key: "a15950433a0d62b8873d6c487c4cbd0c"
server_key_iv: "accfff2d42dc682fd97e8e94ac07b622"
server_key_version: "encryptionkey@0"
I20260806 02:48:07.977805 21654 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:48:07.980329 21676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:07.981594 21654 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:48:07.981729 21654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "98cf6ce0391f4f529d8391a5dd9e4faf"
format_stamp: "Formatted at 2026-08-06 02:48:07 on dist-test-slave-gmjp"
server_key: "a15950433a0d62b8873d6c487c4cbd0c"
server_key_iv: "accfff2d42dc682fd97e8e94ac07b622"
server_key_version: "encryptionkey@0"
I20260806 02:48:07.981840 21654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:07.994462 21654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:07.997957 21654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:07.998186 21654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:08.006037 21654 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:38311
I20260806 02:48:08.006065 21745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:38311 every 8 connection(s)
I20260806 02:48:08.007105 21654 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:48:08.010244 21747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:48:08.014212 21747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38311 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:48:08.016942 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 21654
I20260806 02:48:08.017113 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:48:08.017465 1758 external_mini_cluster.cc:1428] Setting key 8b737a6910274892ad17466256669726
Aug 06 02:48:08 dist-test-slave-gmjp krb5kdc[21635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984487, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.183.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.189@KRBTEST.COM" created.
I20260806 02:48:08.032809 21750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.016797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58374 (local address 127.1.183.190:38311)
0806 02:48:08.017433 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:08.017453 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:08.017629 (+ 176us) server_negotiation.cc:407] Connection header received
0806 02:48:08.018765 (+ 1136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:08.018783 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:08.019043 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:08.019262 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:08.020453 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.021348 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.022976 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.023617 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.023824 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:08.027086 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:08.027093 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:08.027116 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:08.027139 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:08.029206 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:08.029724 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:08.029730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:08.029733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:08.029913 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:08.030276 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:08.030279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:08.030281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:08.030814 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:08.030955 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:08.031348 (+ 393us) server_negotiation.cc:299] Negotiation successful
0806 02:48:08.031627 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":175,"threads_started":1}
I20260806 02:48:08.032806 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.016785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:38311 (local address 127.0.0.1:58374)
0806 02:48:08.017463 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:08.017516 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:08.018402 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:08.019301 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:08.019306 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:08.019801 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:08.020259 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:08.020283 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.021495 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.021502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:08.022812 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:08.022826 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.023863 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.023868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:08.024047 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:08.024811 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:08.024837 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:08.026901 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:08.029334 (+ 2433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:08.029343 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:08.029355 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:08.029600 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:08.030041 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:08.030044 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:08.030046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:08.030174 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:08.030938 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:08.030945 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:08.031095 (+ 150us) client_negotiation.cc:769] Sending connection context
0806 02:48:08.031273 (+ 178us) client_negotiation.cc:240] Negotiation successful
0806 02:48:08.031710 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":113,"thread_start_us":164,"threads_started":1}
I20260806 02:48:08.035394 21747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.183.189" port: 44967 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:48:08.036340 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.035907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:44967 (local address 127.0.0.1:32846)
0806 02:48:08.036086 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:08.036221 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.189:44967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260806 02:48:08.036646 21678 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.183.189:44967: connect: Connection refused (error 111)
I20260806 02:48:08.037674 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.037374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:44967 (local address 127.0.0.1:32860)
0806 02:48:08.037516 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:08.037585 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.189:44967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 02:48:08.037840 21747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.183.189:44967: Network error: Client connection negotiation failed: client connection to 127.1.183.189:44967: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.189.keytab.
Entry for principal kudu/127.1.183.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.189.keytab.
I20260806 02:48:08.049909 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.189:44967
--webserver_interface=127.1.183.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.189.keytab
--principal=kudu/127.1.183.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:41965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:38311,127.1.183.189:44967,127.1.183.188:38519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:48:08.087105 21747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.183.189" port: 44967 } attempt: 1
I20260806 02:48:08.088130 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.087761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:44967 (local address 127.0.0.1:32870)
0806 02:48:08.087948 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:08.088029 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.189:44967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 02:48:08.088941 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.088703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:44967 (local address 127.0.0.1:32886)
0806 02:48:08.088818 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:08.088873 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.189:44967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 02:48:08.089082 21747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.183.189:44967: Network error: Client connection negotiation failed: client connection to 127.1.183.189:44967: connect: Connection refused (error 111)
I20260806 02:48:08.157269 21747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.183.189" port: 44967 } attempt: 2
I20260806 02:48:08.158290 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.157912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:44967 (local address 127.0.0.1:32902)
0806 02:48:08.158100 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:08.158179 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.189:44967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260806 02:48:08.159198 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.158947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:44967 (local address 127.0.0.1:32916)
0806 02:48:08.159089 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:08.159139 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.189:44967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 02:48:08.159354 21747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.183.189:44967: Network error: Client connection negotiation failed: client connection to 127.1.183.189:44967: connect: Connection refused (error 111)
W20260806 02:48:08.175331 21757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:08.175686 21757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:08.175778 21757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:08.180114 21757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:48:08.180286 21757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:08.180310 21757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:08.180348 21757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:48:08.180383 21757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:48:08.186264 21757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:41965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:38311,127.1.183.189:44967,127.1.183.188:38519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.189.keytab
--principal=kudu/127.1.183.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.189:44967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.183.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:48:08.187510 21757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:08.188726 21757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:08.197747 21766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:08.197853 21769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:08.197747 21765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:08.198020 21757 server_base.cc:1034] running on GCE node
I20260806 02:48:08.198681 21757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:08.199990 21757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:08.201220 21757 hybrid_clock.cc:648] HybridClock initialized: now 1785984488201200 us; error 38 us; skew 500 ppm
Aug 06 02:48:08 dist-test-slave-gmjp krb5kdc[21635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984488, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:08.204280 21757 init.cc:377] Logged in from keytab as kudu/127.1.183.189@KRBTEST.COM (short username kudu)
I20260806 02:48:08.205312 21757 webserver.cc:460] Webserver started at http://127.1.183.189:40201/ using document root <none> and password file <none>
I20260806 02:48:08.205935 21757 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:08.206033 21757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:08.206290 21757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:48:08.208575 21757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-1/data/instance:
uuid: "f5a4e3dcdb524a1cb81da8f49c929f68"
format_stamp: "Formatted at 2026-08-06 02:48:08 on dist-test-slave-gmjp"
server_key: "081d89105e55ea1d2269d5a3a91fa920"
server_key_iv: "bc8577dd2e66ab8de4d0a5c7ad20ec29"
server_key_version: "encryptionkey@0"
I20260806 02:48:08.209196 21757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-1/wal/instance:
uuid: "f5a4e3dcdb524a1cb81da8f49c929f68"
format_stamp: "Formatted at 2026-08-06 02:48:08 on dist-test-slave-gmjp"
server_key: "081d89105e55ea1d2269d5a3a91fa920"
server_key_iv: "bc8577dd2e66ab8de4d0a5c7ad20ec29"
server_key_version: "encryptionkey@0"
I20260806 02:48:08.213351 21757 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:48:08.216128 21788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:08.217566 21757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:48:08.217731 21757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-1/wal
uuid: "f5a4e3dcdb524a1cb81da8f49c929f68"
format_stamp: "Formatted at 2026-08-06 02:48:08 on dist-test-slave-gmjp"
server_key: "081d89105e55ea1d2269d5a3a91fa920"
server_key_iv: "bc8577dd2e66ab8de4d0a5c7ad20ec29"
server_key_version: "encryptionkey@0"
I20260806 02:48:08.217864 21757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:08.244530 21757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:08.247684 21757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:08.247925 21757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:08.250532 21747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.183.189" port: 44967 } attempt: 3
I20260806 02:48:08.251318 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.251027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:44967 (local address 127.0.0.1:32924)
0806 02:48:08.251169 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:08.251239 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.189:44967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 02:48:08.252233 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.251873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:44967 (local address 127.0.0.1:32932)
0806 02:48:08.252045 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:08.252107 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.189:44967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260806 02:48:08.252408 21747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.183.189:44967: Network error: Client connection negotiation failed: client connection to 127.1.183.189:44967: connect: Connection refused (error 111)
I20260806 02:48:08.255910 21757 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.189:44967
I20260806 02:48:08.255897 21868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.189:44967 every 8 connection(s)
I20260806 02:48:08.256990 21757 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-1/data/info.pb
I20260806 02:48:08.258461 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 21757
I20260806 02:48:08.258621 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-1/wal/instance
I20260806 02:48:08.258942 1758 external_mini_cluster.cc:1428] Setting key 2237a33a747fc0370843ff898335830a
I20260806 02:48:08.260535 21869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:48:08.264765 21869 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38311 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.183.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.188@KRBTEST.COM" created.
Aug 06 02:48:08 dist-test-slave-gmjp krb5kdc[21635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984488, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.189@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:48:08.280498 21750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.266869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58390 (local address 127.1.183.190:38311)
0806 02:48:08.267021 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:08.267025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:08.268148 (+ 1123us) server_negotiation.cc:407] Connection header received
0806 02:48:08.269202 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:08.269206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:08.269345 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:08.269438 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:08.270740 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.271324 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.272896 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.273337 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.273478 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:08.276394 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:08.276398 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:08.276400 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:08.276428 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:08.278055 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:08.278698 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:08.278702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:08.278706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:08.278792 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:08.279200 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:08.279203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:08.279205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:08.279415 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:08.279527 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:08.280111 (+ 584us) server_negotiation.cc:299] Negotiation successful
0806 02:48:08.280311 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:48:08.281369 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.267412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:38311 (local address 127.0.0.1:58390)
0806 02:48:08.267976 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:08.268022 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:08.268978 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:08.269642 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:08.269650 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:08.269992 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:08.270522 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:08.270538 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.271467 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.271470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:08.272753 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:08.272764 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.273510 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.273513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:08.273665 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:08.274359 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:08.274379 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:08.276123 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:08.278209 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:08.278216 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:08.278236 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:08.278576 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:08.278927 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:08.278930 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:08.278932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:08.279090 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:08.279528 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:08.279537 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:08.279831 (+ 294us) client_negotiation.cc:769] Sending connection context
0806 02:48:08.280034 (+ 203us) client_negotiation.cc:240] Negotiation successful
0806 02:48:08.280469 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":153,"threads_started":1}
I20260806 02:48:08.282963 21869 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.183.189" port: 44967 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:48:08 dist-test-slave-gmjp krb5kdc[21635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984488, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.189@KRBTEST.COM for kudu/127.1.183.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.188.keytab.
Entry for principal kudu/127.1.183.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.188.keytab.
I20260806 02:48:08.292521 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.188:38519
--webserver_interface=127.1.183.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.188.keytab
--principal=kudu/127.1.183.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:41965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:38311,127.1.183.189:44967,127.1.183.188:38519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:48:08.293557 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.283713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32942 (local address 127.1.183.189:44967)
0806 02:48:08.284053 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:08.284062 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:08.284103 (+ 41us) server_negotiation.cc:407] Connection header received
0806 02:48:08.284151 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:08.284159 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:08.284353 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:08.284513 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:08.284867 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.285760 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.286622 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.287009 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.287130 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:08.289430 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:08.289441 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:08.289456 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:08.289477 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:08.291433 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:08.291941 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:08.291946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:08.291947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:08.292031 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:08.292415 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:08.292417 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:08.292419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:08.292741 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:08.292889 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:08.293232 (+ 343us) server_negotiation.cc:299] Negotiation successful
0806 02:48:08.293367 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":156,"threads_started":1}
I20260806 02:48:08.293597 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.283532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:44967 (local address 127.0.0.1:32942)
0806 02:48:08.283665 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:08.283677 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:08.283747 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:08.284489 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:08.284492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:08.284679 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:08.284754 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:08.284759 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.285887 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.285891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:08.286480 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:08.286494 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.287129 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.287132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:08.287220 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:08.287770 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:08.287782 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:08.289256 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:08.291555 (+ 2299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:08.291558 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:08.291560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:08.291813 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:08.292134 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:08.292139 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:08.292143 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:08.292306 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:08.292865 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:08.292893 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:08.293014 (+ 121us) client_negotiation.cc:769] Sending connection context
0806 02:48:08.293140 (+ 126us) client_negotiation.cc:240] Negotiation successful
0806 02:48:08.293316 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 02:48:08.296078 21869 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.183.188" port: 38519 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:48:08.297111 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.296744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:38519 (local address 127.0.0.1:41864)
0806 02:48:08.296930 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:08.297025 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:38519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260806 02:48:08.297384 21791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.183.188:38519: connect: Connection refused (error 111)
I20260806 02:48:08.298372 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.298161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:38519 (local address 127.0.0.1:41866)
0806 02:48:08.298271 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:08.298317 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:38519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 02:48:08.298568 21869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.183.188:38519: Network error: Client connection negotiation failed: client connection to 127.1.183.188:38519: connect: Connection refused (error 111)
I20260806 02:48:08.347908 21869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.183.188" port: 38519 } attempt: 1
I20260806 02:48:08.349159 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.348596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:38519 (local address 127.0.0.1:41874)
0806 02:48:08.348893 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:08.348985 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:38519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260806 02:48:08.349972 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.349721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:38519 (local address 127.0.0.1:41882)
0806 02:48:08.349849 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:08.349912 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:38519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 02:48:08.350154 21869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.183.188:38519: Network error: Client connection negotiation failed: client connection to 127.1.183.188:38519: connect: Connection refused (error 111)
I20260806 02:48:08.395581 21747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.183.189" port: 44967 } attempt: 4
Aug 06 02:48:08 dist-test-slave-gmjp krb5kdc[21635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984487, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for kudu/127.1.183.189@KRBTEST.COM
I20260806 02:48:08.406462 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.396245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:44967 (local address 127.0.0.1:32948)
0806 02:48:08.396468 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:08.396485 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:08.396600 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:08.396909 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:08.396911 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:08.397088 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:08.397595 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:08.397603 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.399203 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.399206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:08.399776 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:08.399786 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.400504 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.400507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:08.400605 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:08.401234 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:08.401250 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:08.402622 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:08.404796 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:08.404798 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:08.404800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:08.405009 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:08.405371 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:08.405374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:08.405376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:08.405473 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:08.405983 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:08.405985 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:08.406088 (+ 103us) client_negotiation.cc:769] Sending connection context
0806 02:48:08.406176 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 02:48:08.406306 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260806 02:48:08.406584 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.396337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32948 (local address 127.1.183.189:44967)
0806 02:48:08.396491 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:08.396495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:08.396572 (+ 77us) server_negotiation.cc:407] Connection header received
0806 02:48:08.396698 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:08.396701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:08.396758 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:08.396872 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:08.397753 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.399043 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.399915 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.400344 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.400492 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:08.402787 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:08.402791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:08.402794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:08.402825 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:08.404633 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:08.405117 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:08.405123 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:08.405125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:08.405222 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:08.405621 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:08.405624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:08.405626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:08.405877 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:08.405988 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:08.406264 (+ 276us) server_negotiation.cc:299] Negotiation successful
0806 02:48:08.406411 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 02:48:08.407409 21747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.183.188" port: 38519 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:48:08.408259 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.407941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:38519 (local address 127.0.0.1:41886)
0806 02:48:08.408092 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:08.408177 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:38519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 02:48:08.408975 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.408734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:38519 (local address 127.0.0.1:41890)
0806 02:48:08.408846 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:08.408905 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:38519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 02:48:08.409173 21747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.183.188:38519: Network error: Client connection negotiation failed: client connection to 127.1.183.188:38519: connect: Connection refused (error 111)
I20260806 02:48:08.418345 21869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.183.188" port: 38519 } attempt: 2
I20260806 02:48:08.419349 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.418933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:38519 (local address 127.0.0.1:41900)
0806 02:48:08.419126 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:08.419220 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:38519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 02:48:08.420226 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.419963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:38519 (local address 127.0.0.1:41914)
0806 02:48:08.420085 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:08.420137 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:38519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 02:48:08.420496 21869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.183.188:38519: Network error: Client connection negotiation failed: client connection to 127.1.183.188:38519: connect: Connection refused (error 111)
W20260806 02:48:08.422645 21878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:08.422968 21878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:08.423071 21878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:08.427292 21878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:48:08.427390 21878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:08.427439 21878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:08.427496 21878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:48:08.427533 21878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:48:08.433274 21878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:41965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:38311,127.1.183.189:44967,127.1.183.188:38519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.188.keytab
--principal=kudu/127.1.183.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.188:38519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.183.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:48:08.434491 21878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:08.435606 21878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:08.444656 21886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:08.444674 21885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:08.444656 21888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:08.444864 21878 server_base.cc:1034] running on GCE node
I20260806 02:48:08.445760 21878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:08.447461 21878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:08.448776 21878 hybrid_clock.cc:648] HybridClock initialized: now 1785984488448751 us; error 41 us; skew 500 ppm
Aug 06 02:48:08 dist-test-slave-gmjp krb5kdc[21635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984488, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:08.451614 21878 init.cc:377] Logged in from keytab as kudu/127.1.183.188@KRBTEST.COM (short username kudu)
I20260806 02:48:08.452577 21878 webserver.cc:460] Webserver started at http://127.1.183.188:44103/ using document root <none> and password file <none>
I20260806 02:48:08.453125 21878 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:08.453184 21878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:08.453359 21878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:48:08.455487 21878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-2/data/instance:
uuid: "4d509f3bd54f49e68b3490a0c15c4e63"
format_stamp: "Formatted at 2026-08-06 02:48:08 on dist-test-slave-gmjp"
server_key: "6433324ad479b0e15b561935f6caba47"
server_key_iv: "fb6811eb01c2238321594fa0873b2b13"
server_key_version: "encryptionkey@0"
I20260806 02:48:08.456024 21878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-2/wal/instance:
uuid: "4d509f3bd54f49e68b3490a0c15c4e63"
format_stamp: "Formatted at 2026-08-06 02:48:08 on dist-test-slave-gmjp"
server_key: "6433324ad479b0e15b561935f6caba47"
server_key_iv: "fb6811eb01c2238321594fa0873b2b13"
server_key_version: "encryptionkey@0"
I20260806 02:48:08.459934 21878 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 02:48:08.462596 21898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:08.463912 21878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 02:48:08.464067 21878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-2/wal
uuid: "4d509f3bd54f49e68b3490a0c15c4e63"
format_stamp: "Formatted at 2026-08-06 02:48:08 on dist-test-slave-gmjp"
server_key: "6433324ad479b0e15b561935f6caba47"
server_key_iv: "fb6811eb01c2238321594fa0873b2b13"
server_key_version: "encryptionkey@0"
I20260806 02:48:08.464267 21878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:08.468353 21747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.183.188" port: 38519 } attempt: 1
I20260806 02:48:08.469293 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.468961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:38519 (local address 127.0.0.1:41930)
0806 02:48:08.469140 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:08.469201 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:38519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 02:48:08.470037 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.469822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:38519 (local address 127.0.0.1:41946)
0806 02:48:08.469935 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:08.469987 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:38519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 02:48:08.470170 21747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.183.188:38519: Network error: Client connection negotiation failed: client connection to 127.1.183.188:38519: connect: Connection refused (error 111)
I20260806 02:48:08.479171 21878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:08.482645 21878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:08.482870 21878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:08.490917 21878 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.188:38519
I20260806 02:48:08.490933 21981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.188:38519 every 8 connection(s)
I20260806 02:48:08.491940 21878 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-2/data/info.pb
I20260806 02:48:08.494906 21984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:48:08.499109 21984 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38311 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:48:08.501760 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 21878
I20260806 02:48:08.501919 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984313887056-1758-0/minicluster-data/master-2/wal/instance
I20260806 02:48:08.502225 1758 external_mini_cluster.cc:1428] Setting key 4e191860fe539acb717c331fdce0906d
Aug 06 02:48:08 dist-test-slave-gmjp krb5kdc[21635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984488, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.188@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
Aug 06 02:48:08 dist-test-slave-gmjp krb5kdc[21635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.188@KRBTEST.COM
I20260806 02:48:08.511691 21869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.183.188" port: 38519 } attempt: 3
I20260806 02:48:08.514313 21750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.501151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58396 (local address 127.1.183.190:38311)
0806 02:48:08.501305 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:08.501309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:08.502217 (+ 908us) server_negotiation.cc:407] Connection header received
0806 02:48:08.503480 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:08.503485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:08.503539 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:08.503615 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:08.504857 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.505367 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.506506 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.506816 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.506906 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:08.509869 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:08.509874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:08.509876 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:08.509908 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:08.511690 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:08.512334 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:08.512337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:08.512339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:08.512448 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:08.512895 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:08.512898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:08.512900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:08.513154 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:08.513221 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:08.513977 (+ 756us) server_negotiation.cc:299] Negotiation successful
0806 02:48:08.514116 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:48:08.515292 21986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.501518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:38311 (local address 127.0.0.1:58396)
0806 02:48:08.502038 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:08.502092 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:08.503225 (+ 1133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:08.503709 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:08.503717 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:08.504195 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:08.504685 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:08.504698 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.505483 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.505486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:08.506390 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:08.506399 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.506958 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.506962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:08.507139 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:08.507854 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:08.507884 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:08.509706 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:08.511815 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:08.511826 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:08.511838 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:08.512218 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:08.512570 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:08.512573 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:08.512575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:08.512771 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:08.513279 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:08.513310 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:08.513643 (+ 333us) client_negotiation.cc:769] Sending connection context
0806 02:48:08.513874 (+ 231us) client_negotiation.cc:240] Negotiation successful
0806 02:48:08.514205 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":23,"thread_start_us":156,"threads_started":1}
I20260806 02:48:08.515475 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.502973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41948 (local address 127.1.183.188:38519)
0806 02:48:08.503348 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:08.503360 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:08.503402 (+ 42us) server_negotiation.cc:407] Connection header received
0806 02:48:08.503613 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:08.503621 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:08.503830 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:08.504022 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:08.504788 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.505819 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.506703 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.507145 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.507244 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:08.510407 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:08.510418 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:08.510430 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:08.510451 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:08.512924 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:08.513423 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:08.513428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:08.513430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:08.513504 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:08.513800 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:08.513805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:08.513806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:08.514113 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:08.514339 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:08.515098 (+ 759us) server_negotiation.cc:299] Negotiation successful
0806 02:48:08.515226 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":283,"spinlock_wait_cycles":2944,"thread_start_us":162,"threads_started":1}
I20260806 02:48:08.517339 21984 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.183.189" port: 44967 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:48:08 dist-test-slave-gmjp krb5kdc[21635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984488, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.189@KRBTEST.COM for kudu/127.1.183.188@KRBTEST.COM
Aug 06 02:48:08 dist-test-slave-gmjp krb5kdc[21635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984488, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.188@KRBTEST.COM for kudu/127.1.183.189@KRBTEST.COM
I20260806 02:48:08.523276 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.512238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:38519 (local address 127.0.0.1:41956)
0806 02:48:08.512492 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:08.512510 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:08.512593 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:08.512863 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:08.512866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:08.513093 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:08.513218 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:08.513226 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.514235 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.514238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:08.515034 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:08.515047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.515973 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.515977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:08.516062 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:08.516624 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:08.516638 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:08.518019 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:08.521478 (+ 3459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:08.521481 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:08.521483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:08.521798 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:08.522142 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:08.522147 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:08.522149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:08.522239 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:08.522734 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:08.522737 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:08.522846 (+ 109us) client_negotiation.cc:769] Sending connection context
0806 02:48:08.522964 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 02:48:08.523102 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260806 02:48:08.523351 21990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.512305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41956 (local address 127.1.183.188:38519)
0806 02:48:08.512650 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:08.512657 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:08.512668 (+ 11us) server_negotiation.cc:407] Connection header received
0806 02:48:08.512697 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:08.512699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:08.512731 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:08.512832 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:08.513344 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.514100 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.515183 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.515830 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.516361 (+ 531us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:08.518170 (+ 1809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:08.518174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:08.518177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:08.518202 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:08.521350 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:08.521930 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:08.521934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:08.521937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:08.522031 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:08.522335 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:08.522340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:08.522342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:08.522633 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:08.522743 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:08.523017 (+ 274us) server_negotiation.cc:299] Negotiation successful
0806 02:48:08.523193 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":140,"threads_started":1}
Aug 06 02:48:08 dist-test-slave-gmjp krb5kdc[21635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:48:08.526947 21869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5a4e3dcdb524a1cb81da8f49c929f68: Bootstrap starting.
I20260806 02:48:08.527154 21986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.517825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:44967 (local address 127.0.0.1:32956)
0806 02:48:08.517977 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:08.517992 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:08.518063 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:08.518406 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:08.518408 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:08.518614 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:08.518748 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:08.518757 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.519808 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.519811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:08.520618 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:08.520628 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.521263 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.521267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:08.521372 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:08.521873 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:08.521907 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:08.524036 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:08.525679 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:08.525686 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:08.525691 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:08.525924 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:08.526188 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:08.526191 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:08.526193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:08.526282 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:08.526660 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:08.526663 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:08.526756 (+ 93us) client_negotiation.cc:769] Sending connection context
0806 02:48:08.526868 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 02:48:08.527001 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260806 02:48:08.527202 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.517913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32956 (local address 127.1.183.189:44967)
0806 02:48:08.518029 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:08.518033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:08.518048 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:48:08.518185 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:08.518190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:08.518236 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:08.518290 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:08.518891 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.519686 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.520729 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.521127 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.521244 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:08.524235 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:08.524239 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:08.524241 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:08.524265 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:08.525563 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:08.526026 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:08.526029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:08.526030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:08.526097 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:08.526373 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:08.526375 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:08.526376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:08.526562 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:08.526640 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:08.526923 (+ 283us) server_negotiation.cc:299] Negotiation successful
0806 02:48:08.527011 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:48:08.529837 21869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5a4e3dcdb524a1cb81da8f49c929f68: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:08.530690 21869 log.cc:824] T 00000000000000000000000000000000 P f5a4e3dcdb524a1cb81da8f49c929f68: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:08.530763 21750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.518145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58412 (local address 127.1.183.190:38311)
0806 02:48:08.518265 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:08.518271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:08.518287 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:48:08.518320 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:08.518322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:08.518364 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:08.518470 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:08.519238 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.519964 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.521023 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.521302 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.521409 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:08.527214 (+ 5805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:08.527221 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:08.527224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:08.527259 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:08.528932 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:08.529401 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:08.529405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:08.529407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:08.529497 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:08.529802 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:08.529806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:08.529808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:08.530061 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:08.530310 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:08.530450 (+ 140us) server_negotiation.cc:299] Negotiation successful
0806 02:48:08.530600 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:48:08.531286 21984 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.183.188" port: 38519 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:48:08.533437 21869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5a4e3dcdb524a1cb81da8f49c929f68: No bootstrap required, opened a new log
Aug 06 02:48:08 dist-test-slave-gmjp krb5kdc[21635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984488, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.188@KRBTEST.COM for kudu/127.1.183.188@KRBTEST.COM
I20260806 02:48:08.537350 21747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.183.188" port: 38519 } attempt: 2
Aug 06 02:48:08 dist-test-slave-gmjp krb5kdc[21635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.189@KRBTEST.COM
I20260806 02:48:08.538022 21869 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5a4e3dcdb524a1cb81da8f49c929f68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cf6ce0391f4f529d8391a5dd9e4faf" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38311 } } peers { permanent_uuid: "f5a4e3dcdb524a1cb81da8f49c929f68" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 44967 } } peers { permanent_uuid: "4d509f3bd54f49e68b3490a0c15c4e63" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 38519 } }
I20260806 02:48:08.538266 21869 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5a4e3dcdb524a1cb81da8f49c929f68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:08.538326 21869 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5a4e3dcdb524a1cb81da8f49c929f68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5a4e3dcdb524a1cb81da8f49c929f68, State: Initialized, Role: FOLLOWER
I20260806 02:48:08.538894 21869 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5a4e3dcdb524a1cb81da8f49c929f68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cf6ce0391f4f529d8391a5dd9e4faf" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38311 } } peers { permanent_uuid: "f5a4e3dcdb524a1cb81da8f49c929f68" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 44967 } } peers { permanent_uuid: "4d509f3bd54f49e68b3490a0c15c4e63" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 38519 } }
I20260806 02:48:08.540633 21869 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5a4e3dcdb524a1cb81da8f49c929f68 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:48:08.540932 21986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.531918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:38519 (local address 127.0.0.1:41966)
0806 02:48:08.532137 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:08.532150 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:08.532269 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:08.532524 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:08.532527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:08.532708 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:08.532777 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:08.532782 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.533526 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.533529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:08.534056 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:08.534063 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.534527 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.534530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:08.534607 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:08.535133 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:08.535147 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:08.536576 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:08.538727 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:08.538734 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:08.538739 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:08.538994 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:08.539295 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:08.539298 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:08.539300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:08.539409 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:08.539874 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:08.539876 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:08.539957 (+ 81us) client_negotiation.cc:769] Sending connection context
0806 02:48:08.540355 (+ 398us) client_negotiation.cc:240] Negotiation successful
0806 02:48:08.540510 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260806 02:48:08.541499 21990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.532223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41966 (local address 127.1.183.188:38519)
0806 02:48:08.532336 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:08.532343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:08.532354 (+ 11us) server_negotiation.cc:407] Connection header received
0806 02:48:08.532382 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:08.532385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:08.532424 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:08.532510 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:08.532900 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.533422 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.534173 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.534417 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.534527 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:08.536743 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:08.536749 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:08.536753 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:08.536786 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:08.538548 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:08.539107 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:08.539109 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:08.539111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:08.539183 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:08.539513 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:08.539516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:08.539518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:08.539772 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:08.541118 (+ 1346us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:08.541194 (+ 76us) server_negotiation.cc:299] Negotiation successful
0806 02:48:08.541308 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 02:48:08.541826 21993 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5a4e3dcdb524a1cb81da8f49c929f68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cf6ce0391f4f529d8391a5dd9e4faf" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38311 } } peers { permanent_uuid: "f5a4e3dcdb524a1cb81da8f49c929f68" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 44967 } } peers { permanent_uuid: "4d509f3bd54f49e68b3490a0c15c4e63" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 38519 } } }
I20260806 02:48:08.541989 21993 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5a4e3dcdb524a1cb81da8f49c929f68 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:48:08.544061 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.531964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32968 (local address 127.1.183.189:44967)
0806 02:48:08.532079 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:08.532084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:08.532135 (+ 51us) server_negotiation.cc:407] Connection header received
0806 02:48:08.532375 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:08.532378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:08.532424 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:08.532474 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:08.533344 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.534042 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.535091 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.535427 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.535499 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:08.539779 (+ 4280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:08.539783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:08.539786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:08.539813 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:08.541482 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:08.542528 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:08.542534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:08.542536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:08.542627 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:08.542975 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:08.542979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:08.542980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:08.543197 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:08.543358 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:08.543772 (+ 414us) server_negotiation.cc:299] Negotiation successful
0806 02:48:08.543896 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Aug 06 02:48:08 dist-test-slave-gmjp krb5kdc[21635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984487, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for kudu/127.1.183.188@KRBTEST.COM
I20260806 02:48:08.547154 21984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63: Bootstrap starting.
W20260806 02:48:08.547247 22009 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f5a4e3dcdb524a1cb81da8f49c929f68: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:48:08.547331 22009 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:48:08.544626 21757 master_runner.cc:186] Error getting master registration for 127.1.183.190:38311: OK, instance_id { permanent_uuid: "98cf6ce0391f4f529d8391a5dd9e4faf" instance_seqno: 1785984488003843 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 02:48:08.548105 21757 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:48:08.549227 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.538124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:38519 (local address 127.0.0.1:41980)
0806 02:48:08.538441 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:08.538470 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:08.538556 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:08.538909 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:08.538915 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:08.539113 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:08.539239 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:08.539251 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.540313 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.540317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:08.541264 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:08.541279 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.542601 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.542604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:08.542711 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:08.543184 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:08.543199 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:08.545484 (+ 2285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:08.547497 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:08.547500 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:08.547502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:08.547755 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:08.548085 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:08.548088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:08.548090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:08.548223 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:08.548692 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:08.548696 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:08.548796 (+ 100us) client_negotiation.cc:769] Sending connection context
0806 02:48:08.548893 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 02:48:08.549050 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220}
I20260806 02:48:08.549773 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.538439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41980 (local address 127.1.183.188:38519)
0806 02:48:08.538580 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:08.538586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:08.538603 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:48:08.538640 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:08.538643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:08.538690 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:08.538764 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:08.539336 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.540136 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.541419 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.541808 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.541918 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:08.545618 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:08.545624 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:08.545629 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:08.545656 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:08.547351 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:08.547888 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:08.547891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:08.547893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:08.547976 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:08.548345 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:08.548348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:08.548350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:08.548574 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:08.549442 (+ 868us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:08.549521 (+ 79us) server_negotiation.cc:299] Negotiation successful
0806 02:48:08.549637 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:48:08.551220 21984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:08.552060 21984 log.cc:824] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:08.553347 21747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98cf6ce0391f4f529d8391a5dd9e4faf: Bootstrap starting.
I20260806 02:48:08.554090 21984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63: No bootstrap required, opened a new log
I20260806 02:48:08.556175 21747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98cf6ce0391f4f529d8391a5dd9e4faf: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:08.556994 1758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:48:08.557137 21747 log.cc:824] T 00000000000000000000000000000000 P 98cf6ce0391f4f529d8391a5dd9e4faf: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:08.557720 21984 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cf6ce0391f4f529d8391a5dd9e4faf" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38311 } } peers { permanent_uuid: "f5a4e3dcdb524a1cb81da8f49c929f68" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 44967 } } peers { permanent_uuid: "4d509f3bd54f49e68b3490a0c15c4e63" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 38519 } }
I20260806 02:48:08.557989 21984 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:08.558040 21984 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d509f3bd54f49e68b3490a0c15c4e63, State: Initialized, Role: FOLLOWER
I20260806 02:48:08.558600 21984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cf6ce0391f4f529d8391a5dd9e4faf" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38311 } } peers { permanent_uuid: "f5a4e3dcdb524a1cb81da8f49c929f68" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 44967 } } peers { permanent_uuid: "4d509f3bd54f49e68b3490a0c15c4e63" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 38519 } }
I20260806 02:48:08.559404 21747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98cf6ce0391f4f529d8391a5dd9e4faf: No bootstrap required, opened a new log
I20260806 02:48:08.559566 22022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cf6ce0391f4f529d8391a5dd9e4faf" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38311 } } peers { permanent_uuid: "f5a4e3dcdb524a1cb81da8f49c929f68" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 44967 } } peers { permanent_uuid: "4d509f3bd54f49e68b3490a0c15c4e63" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 38519 } } }
I20260806 02:48:08.559757 22022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:48:08.560046 21984 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:48:08.564585 21878 master_runner.cc:186] Error getting master registration for 127.1.183.190:38311: OK, instance_id { permanent_uuid: "98cf6ce0391f4f529d8391a5dd9e4faf" instance_seqno: 1785984488003843 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 02:48:08.568218 21878 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:48:08.569001 21747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98cf6ce0391f4f529d8391a5dd9e4faf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cf6ce0391f4f529d8391a5dd9e4faf" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38311 } } peers { permanent_uuid: "f5a4e3dcdb524a1cb81da8f49c929f68" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 44967 } } peers { permanent_uuid: "4d509f3bd54f49e68b3490a0c15c4e63" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 38519 } }
I20260806 02:48:08.569242 21747 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98cf6ce0391f4f529d8391a5dd9e4faf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:08.569286 21747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98cf6ce0391f4f529d8391a5dd9e4faf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98cf6ce0391f4f529d8391a5dd9e4faf, State: Initialized, Role: FOLLOWER
I20260806 02:48:08.569828 21747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98cf6ce0391f4f529d8391a5dd9e4faf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cf6ce0391f4f529d8391a5dd9e4faf" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38311 } } peers { permanent_uuid: "f5a4e3dcdb524a1cb81da8f49c929f68" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 44967 } } peers { permanent_uuid: "4d509f3bd54f49e68b3490a0c15c4e63" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 38519 } }
I20260806 02:48:08.571452 21747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98cf6ce0391f4f529d8391a5dd9e4faf [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:48:08.571950 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.560563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41996 (local address 127.1.183.188:38519)
0806 02:48:08.560668 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:08.560672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:08.562116 (+ 1444us) server_negotiation.cc:407] Connection header received
0806 02:48:08.562334 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:08.562338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:08.562399 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:08.562457 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:08.563586 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.564388 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.565797 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.566130 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.566226 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:08.568187 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:08.568192 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:08.568195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:08.568229 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:08.569863 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:08.570468 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:08.570472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:08.570474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:08.570569 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:08.570868 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:08.570871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:08.570873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:08.571079 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:08.571195 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:08.571628 (+ 433us) server_negotiation.cc:299] Negotiation successful
0806 02:48:08.571750 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260806 02:48:08.567974 22041 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:48:08.573899 22041 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260806 02:48:08.578254 22061 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 98cf6ce0391f4f529d8391a5dd9e4faf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:48:08.578418 22061 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:48:08.578941 21750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.561298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58416 (local address 127.1.183.190:38311)
0806 02:48:08.561429 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:08.561433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:08.564590 (+ 3157us) server_negotiation.cc:407] Connection header received
0806 02:48:08.564695 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:08.564699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:08.564748 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:08.564829 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:08.568543 (+ 3714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.569269 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.570444 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.570737 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.570813 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:08.574544 (+ 3731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:08.574548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:08.574550 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:08.574577 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:08.576268 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:08.576753 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:08.576757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:08.576759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:08.576844 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:08.577414 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:08.577420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:08.577423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:08.577703 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:08.577822 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:08.578624 (+ 802us) server_negotiation.cc:299] Negotiation successful
0806 02:48:08.578768 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 02:48:08.581923 21654 master_runner.cc:263] No leader master found from master 98cf6ce0391f4f529d8391a5dd9e4faf
I20260806 02:48:08.581992 21654 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:48:08.582212 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:08.560320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32972 (local address 127.1.183.189:44967)
0806 02:48:08.560454 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:08.560459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:08.564511 (+ 4052us) server_negotiation.cc:407] Connection header received
0806 02:48:08.564572 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:08.564578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:08.564631 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:08.564708 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:08.570493 (+ 5785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.571261 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.572730 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:08.573015 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:08.573081 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:08.578172 (+ 5091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:08.578176 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:08.578178 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:08.578207 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:08.580360 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:08.580890 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:08.580897 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:08.580898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:08.580993 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:08.581307 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:08.581310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:08.581312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:08.581503 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:08.581643 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:08.581911 (+ 268us) server_negotiation.cc:299] Negotiation successful
0806 02:48:08.582047 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:48:08.582259 22022 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:48:08.582403 22022 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cf6ce0391f4f529d8391a5dd9e4faf" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38311 } } peers { permanent_uuid: "f5a4e3dcdb524a1cb81da8f49c929f68" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 44967 } } peers { permanent_uuid: "4d509f3bd54f49e68b3490a0c15c4e63" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 38519 } }
I20260806 02:48:08.583389 22022 leader_election.cc:288] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98cf6ce0391f4f529d8391a5dd9e4faf (127.1.183.190:38311), f5a4e3dcdb524a1cb81da8f49c929f68 (127.1.183.189:44967)
I20260806 02:48:08.583904 21829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d509f3bd54f49e68b3490a0c15c4e63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5a4e3dcdb524a1cb81da8f49c929f68" is_pre_election: true
I20260806 02:48:08.583887 21713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d509f3bd54f49e68b3490a0c15c4e63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98cf6ce0391f4f529d8391a5dd9e4faf" is_pre_election: true
I20260806 02:48:08.584213 21829 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f5a4e3dcdb524a1cb81da8f49c929f68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d509f3bd54f49e68b3490a0c15c4e63 in term 0.
I20260806 02:48:08.584205 21713 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 98cf6ce0391f4f529d8391a5dd9e4faf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d509f3bd54f49e68b3490a0c15c4e63 in term 0.
I20260806 02:48:08.584648 21900 leader_election.cc:302] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d509f3bd54f49e68b3490a0c15c4e63, f5a4e3dcdb524a1cb81da8f49c929f68; no voters:
I20260806 02:48:08.584836 22043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98cf6ce0391f4f529d8391a5dd9e4faf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cf6ce0391f4f529d8391a5dd9e4faf" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38311 } } peers { permanent_uuid: "f5a4e3dcdb524a1cb81da8f49c929f68" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 44967 } } peers { permanent_uuid: "4d509f3bd54f49e68b3490a0c15c4e63" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 38519 } } }
I20260806 02:48:08.584937 22043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98cf6ce0391f4f529d8391a5dd9e4faf [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:48:08.585297 22022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:48:08.585403 22022 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:48:08.585457 22022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:08.586902 22022 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cf6ce0391f4f529d8391a5dd9e4faf" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38311 } } peers { permanent_uuid: "f5a4e3dcdb524a1cb81da8f49c929f68" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 44967 } } peers { permanent_uuid: "4d509f3bd54f49e68b3490a0c15c4e63" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 38519 } }
I20260806 02:48:08.587273 22022 leader_election.cc:288] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63 [CANDIDATE]: Term 1 election: Requested vote from peers 98cf6ce0391f4f529d8391a5dd9e4faf (127.1.183.190:38311), f5a4e3dcdb524a1cb81da8f49c929f68 (127.1.183.189:44967)
I20260806 02:48:08.587611 21713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d509f3bd54f49e68b3490a0c15c4e63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98cf6ce0391f4f529d8391a5dd9e4faf"
I20260806 02:48:08.587630 21829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d509f3bd54f49e68b3490a0c15c4e63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5a4e3dcdb524a1cb81da8f49c929f68"
I20260806 02:48:08.587754 21829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5a4e3dcdb524a1cb81da8f49c929f68 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:08.587769 21713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98cf6ce0391f4f529d8391a5dd9e4faf [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:08.589231 21829 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f5a4e3dcdb524a1cb81da8f49c929f68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d509f3bd54f49e68b3490a0c15c4e63 in term 1.
I20260806 02:48:08.589602 21713 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 98cf6ce0391f4f529d8391a5dd9e4faf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d509f3bd54f49e68b3490a0c15c4e63 in term 1.
I20260806 02:48:08.589825 21900 leader_election.cc:302] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d509f3bd54f49e68b3490a0c15c4e63, f5a4e3dcdb524a1cb81da8f49c929f68; no voters:
I20260806 02:48:08.590029 22022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:48:08.590265 22022 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63 [term 1 LEADER]: Becoming Leader. State: Replica: 4d509f3bd54f49e68b3490a0c15c4e63, State: Running, Role: LEADER
I20260806 02:48:08.590691 22022 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cf6ce0391f4f529d8391a5dd9e4faf" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38311 } } peers { permanent_uuid: "f5a4e3dcdb524a1cb81da8f49c929f68" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 44967 } } peers { permanent_uuid: "4d509f3bd54f49e68b3490a0c15c4e63" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 38519 } }
I20260806 02:48:08.593253 22063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d509f3bd54f49e68b3490a0c15c4e63. Latest consensus state: current_term: 1 leader_uuid: "4d509f3bd54f49e68b3490a0c15c4e63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cf6ce0391f4f529d8391a5dd9e4faf" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38311 } } peers { permanent_uuid: "f5a4e3dcdb524a1cb81da8f49c929f68" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 44967 } } peers { permanent_uuid: "4d509f3bd54f49e68b3490a0c15c4e63" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 38519 } } }
I20260806 02:48:08.593391 22063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63 [sys.catalog]: This master's current role is: LEADER
I20260806 02:48:08.593792 22065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:48:08.594748 22065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:48:08.600764 21713 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 98cf6ce0391f4f529d8391a5dd9e4faf [term 1 FOLLOWER]: Refusing update from remote peer 4d509f3bd54f49e68b3490a0c15c4e63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:48:08.600800 21829 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f5a4e3dcdb524a1cb81da8f49c929f68 [term 1 FOLLOWER]: Refusing update from remote peer 4d509f3bd54f49e68b3490a0c15c4e63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:48:08.601435 22022 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98cf6ce0391f4f529d8391a5dd9e4faf" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:48:08.601588 22063 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5a4e3dcdb524a1cb81da8f49c929f68" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 44967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:48:08.606045 21993 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5a4e3dcdb524a1cb81da8f49c929f68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d509f3bd54f49e68b3490a0c15c4e63. Latest consensus state: current_term: 1 leader_uuid: "4d509f3bd54f49e68b3490a0c15c4e63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cf6ce0391f4f529d8391a5dd9e4faf" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38311 } } peers { permanent_uuid: "f5a4e3dcdb524a1cb81da8f49c929f68" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 44967 } } peers { permanent_uuid: "4d509f3bd54f49e68b3490a0c15c4e63" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 38519 } } }
I20260806 02:48:08.606182 21993 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5a4e3dcdb524a1cb81da8f49c929f68 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:48:08.606772 22043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98cf6ce0391f4f529d8391a5dd9e4faf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d509f3bd54f49e68b3490a0c15c4e63. Latest consensus state: current_term: 1 leader_uuid: "4d509f3bd54f49e68b3490a0c15c4e63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cf6ce0391f4f529d8391a5dd9e4faf" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38311 } } peers { permanent_uuid: "f5a4e3dcdb524a1cb81da8f49c929f68" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 44967 } } peers { permanent_uuid: "4d509f3bd54f49e68b3490a0c15c4e63" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 38519 } } }
I20260806 02:48:08.606897 22043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98cf6ce0391f4f529d8391a5dd9e4faf [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:48:08.617349 22066 mvcc.cc:205] Tried to move back new op lower bound from 7315392465297346560 to 7315392465270476800. Current Snapshot: MvccSnapshot[applied={T|T < 7315392465297346560}]
I20260806 02:48:08.619092 22022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4d509f3bd54f49e68b3490a0c15c4e63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cf6ce0391f4f529d8391a5dd9e4faf" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38311 } } peers { permanent_uuid: "f5a4e3dcdb524a1cb81da8f49c929f68" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 44967 } } peers { permanent_uuid: "4d509f3bd54f49e68b3490a0c15c4e63" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 38519 } } }
I20260806 02:48:08.619236 22022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63 [sys.catalog]: This master's current role is: LEADER
I20260806 02:48:08.620555 21993 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5a4e3dcdb524a1cb81da8f49c929f68 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4d509f3bd54f49e68b3490a0c15c4e63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cf6ce0391f4f529d8391a5dd9e4faf" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38311 } } peers { permanent_uuid: "f5a4e3dcdb524a1cb81da8f49c929f68" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 44967 } } peers { permanent_uuid: "4d509f3bd54f49e68b3490a0c15c4e63" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 38519 } } }
I20260806 02:48:08.620679 21993 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5a4e3dcdb524a1cb81da8f49c929f68 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:48:08.621203 22070 mvcc.cc:205] Tried to move back new op lower bound from 7315392465297346560 to 7315392465270476800. Current Snapshot: MvccSnapshot[applied={T|T < 7315392465297346560}]
I20260806 02:48:08.625622 22068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4d509f3bd54f49e68b3490a0c15c4e63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cf6ce0391f4f529d8391a5dd9e4faf" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38311 } } peers { permanent_uuid: "f5a4e3dcdb524a1cb81da8f49c929f68" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 44967 } } peers { permanent_uuid: "4d509f3bd54f49e68b3490a0c15c4e63" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 38519 } } }
I20260806 02:48:08.625782 22068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63 [sys.catalog]: This master's current role is: LEADER
I20260806 02:48:08.627023 22043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98cf6ce0391f4f529d8391a5dd9e4faf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4d509f3bd54f49e68b3490a0c15c4e63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cf6ce0391f4f529d8391a5dd9e4faf" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38311 } } peers { permanent_uuid: "f5a4e3dcdb524a1cb81da8f49c929f68" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 44967 } } peers { permanent_uuid: "4d509f3bd54f49e68b3490a0c15c4e63" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 38519 } } }
I20260806 02:48:08.627151 22043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98cf6ce0391f4f529d8391a5dd9e4faf [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:48:08.627398 22069 mvcc.cc:205] Tried to move back new op lower bound from 7315392465297346560 to 7315392465270476800. Current Snapshot: MvccSnapshot[applied={T|T < 7315392465297346560}]
I20260806 02:48:08.627888 22065 catalog_manager.cc:1347] Generated new cluster ID: f10696f31de64713825fbfa83216935b
I20260806 02:48:08.627956 22065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:48:08.671743 22065 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:48:08.673306 22065 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:48:08.693997 22065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d509f3bd54f49e68b3490a0c15c4e63: Generated new TSK 0
I20260806 02:48:08.697513 22065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:48:09.548817 22009 catalog_manager.cc:1259] Loaded cluster ID: f10696f31de64713825fbfa83216935b
I20260806 02:48:09.548924 22009 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f5a4e3dcdb524a1cb81da8f49c929f68: loading cluster ID for follower catalog manager: success
I20260806 02:48:09.550601 22009 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f5a4e3dcdb524a1cb81da8f49c929f68: acquiring CA information for follower catalog manager: success
I20260806 02:48:09.551776 22009 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f5a4e3dcdb524a1cb81da8f49c929f68: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:48:09.580190 22061 catalog_manager.cc:1259] Loaded cluster ID: f10696f31de64713825fbfa83216935b
I20260806 02:48:09.580317 22061 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 98cf6ce0391f4f529d8391a5dd9e4faf: loading cluster ID for follower catalog manager: success
I20260806 02:48:09.582064 22061 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 98cf6ce0391f4f529d8391a5dd9e4faf: acquiring CA information for follower catalog manager: success
I20260806 02:48:09.583349 22061 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 98cf6ce0391f4f529d8391a5dd9e4faf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:48:09.773313 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 21654
I20260806 02:48:09.785508 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 21757
I20260806 02:48:09.797824 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 21878
2026-08-06T02:48:09Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3362 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:48:09 dist-test-slave-gmjp krb5kdc[22088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:48:09 dist-test-slave-gmjp krb5kdc[22088](info): set up 2 sockets
Aug 06 02:48:09 dist-test-slave-gmjp krb5kdc[22088](info): commencing operation
krb5kdc: starting...
W20260806 02:48:11.028743 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.203s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:48:11 dist-test-slave-gmjp krb5kdc[22088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:48:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:48:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
I20260806 02:48:11.167546 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:39205
--webserver_interface=127.1.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:35795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:39205
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:48:11.288877 22104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:11.289171 22104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:11.289213 22104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:11.293447 22104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:48:11.293530 22104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:11.293550 22104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:11.293567 22104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:48:11.293583 22104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:48:11.299225 22104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:35795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:39205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:39205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:48:11.300498 22104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:11.301604 22104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:11.310967 22110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:11.311164 22104 server_base.cc:1034] running on GCE node
W20260806 02:48:11.310950 22112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:11.310953 22109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:11.311937 22104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:11.313117 22104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:11.314280 22104 hybrid_clock.cc:648] HybridClock initialized: now 1785984491314250 us; error 44 us; skew 500 ppm
Aug 06 02:48:11 dist-test-slave-gmjp krb5kdc[22088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984491, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:11.319980 22104 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:48:11.321026 22104 webserver.cc:460] Webserver started at http://127.1.183.190:46047/ using document root <none> and password file <none>
I20260806 02:48:11.321574 22104 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:11.321637 22104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:11.321808 22104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:48:11.323904 22104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "9e22c7a1fe6044a4ae7921d2eef4a899"
format_stamp: "Formatted at 2026-08-06 02:48:11 on dist-test-slave-gmjp"
server_key: "bf3660ea429ea005a4671c8dc4e67025"
server_key_iv: "9a6bb6ea849bfcf0fe0f7ea1ce22996a"
server_key_version: "encryptionkey@0"
I20260806 02:48:11.324502 22104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "9e22c7a1fe6044a4ae7921d2eef4a899"
format_stamp: "Formatted at 2026-08-06 02:48:11 on dist-test-slave-gmjp"
server_key: "bf3660ea429ea005a4671c8dc4e67025"
server_key_iv: "9a6bb6ea849bfcf0fe0f7ea1ce22996a"
server_key_version: "encryptionkey@0"
I20260806 02:48:11.328395 22104 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:48:11.330823 22122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:11.332108 22104 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:48:11.332283 22104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "9e22c7a1fe6044a4ae7921d2eef4a899"
format_stamp: "Formatted at 2026-08-06 02:48:11 on dist-test-slave-gmjp"
server_key: "bf3660ea429ea005a4671c8dc4e67025"
server_key_iv: "9a6bb6ea849bfcf0fe0f7ea1ce22996a"
server_key_version: "encryptionkey@0"
I20260806 02:48:11.332389 22104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:11.353632 22104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:11.357266 22104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:11.357445 22104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:11.364938 22104 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:39205
I20260806 02:48:11.364956 22201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:39205 every 8 connection(s)
I20260806 02:48:11.365940 22104 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:48:11.368985 22202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:48:11.374878 22202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e22c7a1fe6044a4ae7921d2eef4a899: Bootstrap starting.
I20260806 02:48:11.375830 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 22104
I20260806 02:48:11.375988 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:48:11.376360 1758 external_mini_cluster.cc:1428] Setting key 951c4ac068b48a2f8e4d36a7eecc5a0f
I20260806 02:48:11.378171 22202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e22c7a1fe6044a4ae7921d2eef4a899: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:11.378997 22202 log.cc:824] T 00000000000000000000000000000000 P 9e22c7a1fe6044a4ae7921d2eef4a899: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:11.381358 22202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e22c7a1fe6044a4ae7921d2eef4a899: No bootstrap required, opened a new log
Aug 06 02:48:11 dist-test-slave-gmjp krb5kdc[22088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:48:11.386332 22202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e22c7a1fe6044a4ae7921d2eef4a899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e22c7a1fe6044a4ae7921d2eef4a899" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39205 } }
I20260806 02:48:11.386633 22202 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e22c7a1fe6044a4ae7921d2eef4a899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:11.386687 22202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e22c7a1fe6044a4ae7921d2eef4a899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e22c7a1fe6044a4ae7921d2eef4a899, State: Initialized, Role: FOLLOWER
I20260806 02:48:11.387230 22202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e22c7a1fe6044a4ae7921d2eef4a899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e22c7a1fe6044a4ae7921d2eef4a899" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39205 } }
I20260806 02:48:11.387442 22202 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e22c7a1fe6044a4ae7921d2eef4a899 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:48:11.387534 22202 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e22c7a1fe6044a4ae7921d2eef4a899 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:48:11.387655 22202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e22c7a1fe6044a4ae7921d2eef4a899 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:11.389068 22202 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e22c7a1fe6044a4ae7921d2eef4a899 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e22c7a1fe6044a4ae7921d2eef4a899" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39205 } }
I20260806 02:48:11.389495 22202 leader_election.cc:302] T 00000000000000000000000000000000 P 9e22c7a1fe6044a4ae7921d2eef4a899 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e22c7a1fe6044a4ae7921d2eef4a899; no voters:
I20260806 02:48:11.389976 22202 leader_election.cc:288] T 00000000000000000000000000000000 P 9e22c7a1fe6044a4ae7921d2eef4a899 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:48:11.390115 22209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e22c7a1fe6044a4ae7921d2eef4a899 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:48:11.390434 22209 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e22c7a1fe6044a4ae7921d2eef4a899 [term 1 LEADER]: Becoming Leader. State: Replica: 9e22c7a1fe6044a4ae7921d2eef4a899, State: Running, Role: LEADER
I20260806 02:48:11.390837 22209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e22c7a1fe6044a4ae7921d2eef4a899 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e22c7a1fe6044a4ae7921d2eef4a899" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39205 } }
I20260806 02:48:11.391618 22202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e22c7a1fe6044a4ae7921d2eef4a899 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:48:11.392911 22207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:11.377584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34772 (local address 127.1.183.190:39205)
0806 02:48:11.378055 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:11.378070 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:11.378106 (+ 36us) server_negotiation.cc:407] Connection header received
0806 02:48:11.378848 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:11.378875 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:11.379274 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:11.379665 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:11.380586 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:11.381661 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:11.382843 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:11.383349 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:11.383475 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:11.386692 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:11.386702 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:11.386724 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:11.386769 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:11.389046 (+ 2277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:11.389612 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:11.389621 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:11.389627 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:11.389816 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:11.390267 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:11.390274 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:11.390278 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:11.390831 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:11.391085 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:11.391421 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 02:48:11.391759 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":147,"threads_started":1}
I20260806 02:48:11.393044 22213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e22c7a1fe6044a4ae7921d2eef4a899 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e22c7a1fe6044a4ae7921d2eef4a899. Latest consensus state: current_term: 1 leader_uuid: "9e22c7a1fe6044a4ae7921d2eef4a899" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e22c7a1fe6044a4ae7921d2eef4a899" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39205 } } }
I20260806 02:48:11.393343 22213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e22c7a1fe6044a4ae7921d2eef4a899 [sys.catalog]: This master's current role is: LEADER
I20260806 02:48:11.393045 22211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e22c7a1fe6044a4ae7921d2eef4a899 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e22c7a1fe6044a4ae7921d2eef4a899" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e22c7a1fe6044a4ae7921d2eef4a899" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 39205 } } }
I20260806 02:48:11.393579 22211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e22c7a1fe6044a4ae7921d2eef4a899 [sys.catalog]: This master's current role is: LEADER
I20260806 02:48:11.393903 22221 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:48:11.396641 1758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:48:11.397328 22221 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:48:11.403611 22221 catalog_manager.cc:1347] Generated new cluster ID: 60cf4236db9645929fed24aa80ae4200
I20260806 02:48:11.403716 22221 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:48:11.414691 22221 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:48:11.415764 22221 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:48:11.424201 22221 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e22c7a1fe6044a4ae7921d2eef4a899: Generated new TSK 0
I20260806 02:48:11.425014 22221 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:48:11.453058 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 22104
2026-08-06T02:48:11Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1653 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:48:11 dist-test-slave-gmjp krb5kdc[22240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:48:11 dist-test-slave-gmjp krb5kdc[22240](info): set up 2 sockets
Aug 06 02:48:11 dist-test-slave-gmjp krb5kdc[22240](info): commencing operation
krb5kdc: starting...
W20260806 02:48:12.706794 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.227s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:48:12 dist-test-slave-gmjp krb5kdc[22240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:48:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:48:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
I20260806 02:48:12.843892 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:46301
--webserver_interface=127.1.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:38541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:46301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:48:12.962077 22262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:12.962430 22262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:12.962513 22262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:12.966717 22262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:48:12.966830 22262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:12.966868 22262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:12.966907 22262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:48:12.966955 22262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:48:12.972890 22262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:38541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:46301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:46301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:48:12.974161 22262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:12.975304 22262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:12.984417 22270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:12.984421 22272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:12.984617 22262 server_base.cc:1034] running on GCE node
W20260806 02:48:12.984843 22277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:12.985491 22262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:12.986641 22262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:12.987851 22262 hybrid_clock.cc:648] HybridClock initialized: now 1785984492987833 us; error 33 us; skew 500 ppm
Aug 06 02:48:12 dist-test-slave-gmjp krb5kdc[22240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984492, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:12.990908 22262 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:48:12.991858 22262 webserver.cc:460] Webserver started at http://127.1.183.190:43579/ using document root <none> and password file <none>
I20260806 02:48:12.992512 22262 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:12.992607 22262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:12.992851 22262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:48:12.994971 22262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "efc9d63000364ad9a38827aededd20e2"
format_stamp: "Formatted at 2026-08-06 02:48:12 on dist-test-slave-gmjp"
server_key: "4526a6d2f8b50564bde8aae1fcdf5fb7"
server_key_iv: "9b3c512375fd98816072daaee70c5ec7"
server_key_version: "encryptionkey@0"
I20260806 02:48:12.995576 22262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "efc9d63000364ad9a38827aededd20e2"
format_stamp: "Formatted at 2026-08-06 02:48:12 on dist-test-slave-gmjp"
server_key: "4526a6d2f8b50564bde8aae1fcdf5fb7"
server_key_iv: "9b3c512375fd98816072daaee70c5ec7"
server_key_version: "encryptionkey@0"
I20260806 02:48:12.999423 22262 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260806 02:48:13.002032 22286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:13.003298 22262 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:48:13.003454 22262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "efc9d63000364ad9a38827aededd20e2"
format_stamp: "Formatted at 2026-08-06 02:48:12 on dist-test-slave-gmjp"
server_key: "4526a6d2f8b50564bde8aae1fcdf5fb7"
server_key_iv: "9b3c512375fd98816072daaee70c5ec7"
server_key_version: "encryptionkey@0"
I20260806 02:48:13.003574 22262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:13.034214 22262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:13.038048 22262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:13.038301 22262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:13.047925 22262 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:46301
I20260806 02:48:13.048031 22358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:46301 every 8 connection(s)
I20260806 02:48:13.049059 22262 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:48:13.051855 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 22262
I20260806 02:48:13.052004 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:48:13.052080 22359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:48:13.052397 1758 external_mini_cluster.cc:1428] Setting key 6f0c8cf8d29f2f4e97c280cbd6f5759d
I20260806 02:48:13.059175 22359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2: Bootstrap starting.
Aug 06 02:48:13 dist-test-slave-gmjp krb5kdc[22240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:48:13.062183 22359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:13.062933 22359 log.cc:824] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:13.064972 22359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2: No bootstrap required, opened a new log
I20260806 02:48:13.067536 22365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:13.053928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40532 (local address 127.1.183.190:46301)
0806 02:48:13.054362 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:13.054378 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:13.054414 (+ 36us) server_negotiation.cc:407] Connection header received
0806 02:48:13.055149 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:13.055167 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:13.055513 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:13.055818 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:13.056878 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:13.057899 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:13.059088 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:13.059616 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:13.059735 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:13.062639 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:13.062644 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:13.062659 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:13.062687 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:13.064546 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:13.065019 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:13.065024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:13.065029 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:13.065182 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:13.065473 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:13.065477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:13.065479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:13.066005 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:13.066203 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:13.066503 (+ 300us) server_negotiation.cc:299] Negotiation successful
0806 02:48:13.066809 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":143,"threads_started":1}
I20260806 02:48:13.068996 22359 raft_consensus.cc:348] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc9d63000364ad9a38827aededd20e2" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 46301 } }
I20260806 02:48:13.069253 22359 raft_consensus.cc:374] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:13.069335 22359 raft_consensus.cc:729] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efc9d63000364ad9a38827aededd20e2, State: Initialized, Role: FOLLOWER
I20260806 02:48:13.069845 22359 consensus_queue.cc:260] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc9d63000364ad9a38827aededd20e2" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 46301 } }
I20260806 02:48:13.070065 22359 raft_consensus.cc:388] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:48:13.070164 22359 raft_consensus.cc:482] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:48:13.070297 22359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:13.071267 22359 raft_consensus.cc:504] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc9d63000364ad9a38827aededd20e2" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 46301 } }
I20260806 02:48:13.071666 22359 leader_election.cc:302] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efc9d63000364ad9a38827aededd20e2; no voters:
I20260806 02:48:13.072059 22359 leader_election.cc:288] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:48:13.072189 22368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:48:13.072449 22368 raft_consensus.cc:686] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [term 1 LEADER]: Becoming Leader. State: Replica: efc9d63000364ad9a38827aededd20e2, State: Running, Role: LEADER
I20260806 02:48:13.072829 22368 consensus_queue.cc:237] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc9d63000364ad9a38827aededd20e2" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 46301 } }
I20260806 02:48:13.073408 22359 sys_catalog.cc:564] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:48:13.074671 22371 sys_catalog.cc:455] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efc9d63000364ad9a38827aededd20e2. Latest consensus state: current_term: 1 leader_uuid: "efc9d63000364ad9a38827aededd20e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc9d63000364ad9a38827aededd20e2" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 46301 } } }
I20260806 02:48:13.074673 22369 sys_catalog.cc:455] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efc9d63000364ad9a38827aededd20e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc9d63000364ad9a38827aededd20e2" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 46301 } } }
I20260806 02:48:13.074831 22371 sys_catalog.cc:458] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [sys.catalog]: This master's current role is: LEADER
I20260806 02:48:13.074905 22369 sys_catalog.cc:458] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [sys.catalog]: This master's current role is: LEADER
I20260806 02:48:13.076110 22382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:48:13.078919 22382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:48:13.084471 22382 catalog_manager.cc:1347] Generated new cluster ID: 7ca99d4d402d42278037be43e9849523
I20260806 02:48:13.084623 22382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:48:13.101264 22382 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:48:13.102365 22382 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:48:13.112488 22382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2: Generated new TSK 0
I20260806 02:48:13.113309 22382 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
I20260806 02:48:13.153467 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.129:0
--local_ip_for_outbound_sockets=127.1.183.129
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:46301
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:38541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:48:13.276790 22395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:13.277073 22395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:13.277159 22395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:13.281411 22395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:13.281476 22395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:13.281553 22395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.129
I20260806 02:48:13.287025 22395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:38541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:46301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.129
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:48:13.288244 22395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:13.289392 22395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:13.300047 22403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:13.300047 22405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:13.300051 22407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:13.300237 22395 server_base.cc:1034] running on GCE node
I20260806 02:48:13.300853 22395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:13.301575 22395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:13.302770 22395 hybrid_clock.cc:648] HybridClock initialized: now 1785984493302756 us; error 25 us; skew 500 ppm
Aug 06 02:48:13 dist-test-slave-gmjp krb5kdc[22240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984493, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:13.305729 22395 init.cc:377] Logged in from keytab as kudu/127.1.183.129@KRBTEST.COM (short username kudu)
I20260806 02:48:13.306840 22395 webserver.cc:460] Webserver started at http://127.1.183.129:40849/ using document root <none> and password file <none>
I20260806 02:48:13.307524 22395 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:13.307628 22395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:13.307933 22395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:48:13.310256 22395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/data/instance:
uuid: "ebe3e718f749433d879f26d19ec8e977"
format_stamp: "Formatted at 2026-08-06 02:48:13 on dist-test-slave-gmjp"
server_key: "dc2ec2dde1a60ae694d399cc8137925d"
server_key_iv: "35ab59c11508b49a521ad83ac33ad786"
server_key_version: "encryptionkey@0"
I20260806 02:48:13.310894 22395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance:
uuid: "ebe3e718f749433d879f26d19ec8e977"
format_stamp: "Formatted at 2026-08-06 02:48:13 on dist-test-slave-gmjp"
server_key: "dc2ec2dde1a60ae694d399cc8137925d"
server_key_iv: "35ab59c11508b49a521ad83ac33ad786"
server_key_version: "encryptionkey@0"
I20260806 02:48:13.314978 22395 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:48:13.317698 22418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:13.319068 22395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:48:13.319216 22395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/wal
uuid: "ebe3e718f749433d879f26d19ec8e977"
format_stamp: "Formatted at 2026-08-06 02:48:13 on dist-test-slave-gmjp"
server_key: "dc2ec2dde1a60ae694d399cc8137925d"
server_key_iv: "35ab59c11508b49a521ad83ac33ad786"
server_key_version: "encryptionkey@0"
I20260806 02:48:13.319339 22395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:13.338937 22395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:13.342281 22395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:13.342515 22395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:13.343173 22395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:48:13.344065 22395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:48:13.344141 22395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:13.344228 22395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:48:13.344275 22395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:13.353757 22395 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.129:40057
I20260806 02:48:13.353780 22579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.129:40057 every 8 connection(s)
I20260806 02:48:13.354735 22395 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
I20260806 02:48:13.362133 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 22395
I20260806 02:48:13.362265 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance
I20260806 02:48:13.362594 1758 external_mini_cluster.cc:1428] Setting key f604e8f7cb8c20ccbef9b3e6ab1db877
Aug 06 02:48:13 dist-test-slave-gmjp krb5kdc[22240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984493, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:48:13.370306 22365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:13.356757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:39493 (local address 127.1.183.190:46301)
0806 02:48:13.356913 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:13.356917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:13.357793 (+ 876us) server_negotiation.cc:407] Connection header received
0806 02:48:13.359196 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:13.359200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:13.359266 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:13.359384 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:13.360698 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:13.361255 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:13.362595 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:13.362896 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:13.363020 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:13.365804 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:13.365807 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:13.365809 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:13.365836 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:13.367646 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:13.368407 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:13.368410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:13.368411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:13.368498 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:13.368942 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:13.368945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:13.368946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:13.369138 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:13.369215 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:13.369956 (+ 741us) server_negotiation.cc:299] Negotiation successful
0806 02:48:13.370078 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 02:48:13.371621 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:13.357145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.129:39493)
0806 02:48:13.357627 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:13.357672 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:13.358963 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:13.359589 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:13.359596 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:13.359983 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:13.360525 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:13.360540 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:13.361400 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:13.361411 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:13.362459 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:13.362470 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:13.363085 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:13.363090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:13.363276 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:13.363944 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:13.363972 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:13.365602 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:13.367801 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:13.367807 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:13.367822 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:13.368303 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:13.368600 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:13.368605 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:13.368610 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:13.368818 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:13.369248 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:13.369258 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:13.369633 (+ 375us) client_negotiation.cc:769] Sending connection context
0806 02:48:13.369916 (+ 283us) client_negotiation.cc:240] Negotiation successful
0806 02:48:13.370339 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":115,"threads_started":1}
I20260806 02:48:13.373245 22580 heartbeater.cc:347] Connected to a master server at 127.1.183.190:46301
I20260806 02:48:13.373535 22580 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:13.374145 22580 heartbeater.cc:511] Master 127.1.183.190:46301 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.183.130@KRBTEST.COM; defaulting to no policy
I20260806 02:48:13.375993 22309 ts_manager.cc:194] Registered new tserver with Master: ebe3e718f749433d879f26d19ec8e977 (127.1.183.129:40057)
I20260806 02:48:13.377363 22309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.129@KRBTEST.COM'} at 127.1.183.129:39493
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
I20260806 02:48:13.395751 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.130:0
--local_ip_for_outbound_sockets=127.1.183.130
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:46301
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:38541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:48:13.516141 22589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:13.516563 22589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:13.516667 22589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:13.520924 22589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:13.521029 22589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:13.521124 22589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.130
I20260806 02:48:13.526909 22589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:38541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:46301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.130
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:48:13.528190 22589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:13.529304 22589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:13.539601 22607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:13.539615 22605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:13.539665 22601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:13.539971 22589 server_base.cc:1034] running on GCE node
I20260806 02:48:13.540552 22589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:13.541254 22589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:13.542457 22589 hybrid_clock.cc:648] HybridClock initialized: now 1785984493542426 us; error 50 us; skew 500 ppm
Aug 06 02:48:13 dist-test-slave-gmjp krb5kdc[22240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984493, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:13.545451 22589 init.cc:377] Logged in from keytab as kudu/127.1.183.130@KRBTEST.COM (short username kudu)
I20260806 02:48:13.546377 22589 webserver.cc:460] Webserver started at http://127.1.183.130:39981/ using document root <none> and password file <none>
I20260806 02:48:13.546939 22589 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:13.547000 22589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:13.547176 22589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:48:13.549361 22589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/data/instance:
uuid: "5d733086cd7249c08b869c31d708b865"
format_stamp: "Formatted at 2026-08-06 02:48:13 on dist-test-slave-gmjp"
server_key: "38a71d57c4f6d1046ac832ef7dabeb58"
server_key_iv: "e72337064f96990dc798ce17925891cf"
server_key_version: "encryptionkey@0"
I20260806 02:48:13.549909 22589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance:
uuid: "5d733086cd7249c08b869c31d708b865"
format_stamp: "Formatted at 2026-08-06 02:48:13 on dist-test-slave-gmjp"
server_key: "38a71d57c4f6d1046ac832ef7dabeb58"
server_key_iv: "e72337064f96990dc798ce17925891cf"
server_key_version: "encryptionkey@0"
I20260806 02:48:13.553977 22589 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260806 02:48:13.556641 22617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:13.557971 22589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:48:13.558092 22589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/wal
uuid: "5d733086cd7249c08b869c31d708b865"
format_stamp: "Formatted at 2026-08-06 02:48:13 on dist-test-slave-gmjp"
server_key: "38a71d57c4f6d1046ac832ef7dabeb58"
server_key_iv: "e72337064f96990dc798ce17925891cf"
server_key_version: "encryptionkey@0"
I20260806 02:48:13.558193 22589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:13.579495 22589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:13.582901 22589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:13.583076 22589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:13.583686 22589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:48:13.584723 22589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:48:13.584777 22589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:13.584823 22589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:48:13.584877 22589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:13.594386 22589 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.130:37461
I20260806 02:48:13.594410 22781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.130:37461 every 8 connection(s)
I20260806 02:48:13.595453 22589 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
I20260806 02:48:13.604439 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 22589
I20260806 02:48:13.604585 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance
I20260806 02:48:13.604889 1758 external_mini_cluster.cc:1428] Setting key 128d377deedcfb2e40e218c55781c172
Aug 06 02:48:13 dist-test-slave-gmjp krb5kdc[22240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984493, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:48:13.610595 22365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:13.597463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:42263 (local address 127.1.183.190:46301)
0806 02:48:13.597643 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:13.597648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:13.598576 (+ 928us) server_negotiation.cc:407] Connection header received
0806 02:48:13.599881 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:13.599883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:13.599942 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:13.600024 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:13.601333 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:13.601862 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:13.603050 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:13.603264 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:13.603347 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:13.606478 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:13.606483 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:13.606485 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:13.606519 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:13.608284 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:13.608903 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:13.608907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:13.608909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:13.609002 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:13.609384 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:13.609389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:13.609391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:13.609605 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:13.609717 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:13.610283 (+ 566us) server_negotiation.cc:299] Negotiation successful
0806 02:48:13.610420 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 02:48:13.611842 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:13.597897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.130:42263)
0806 02:48:13.598397 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:13.598443 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:13.599668 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:13.600266 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:13.600273 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:13.600664 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:13.601187 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:13.601202 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:13.602004 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:13.602007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:13.602935 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:13.602945 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:13.603407 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:13.603410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:13.603554 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:13.604279 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:13.604297 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:13.606287 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:13.608430 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:13.608439 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:13.608461 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:13.608774 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:13.609105 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:13.609108 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:13.609110 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:13.609281 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:13.609719 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:13.609729 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:13.610026 (+ 297us) client_negotiation.cc:769] Sending connection context
0806 02:48:13.610238 (+ 212us) client_negotiation.cc:240] Negotiation successful
0806 02:48:13.610643 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":137,"threads_started":1}
I20260806 02:48:13.613248 22783 heartbeater.cc:347] Connected to a master server at 127.1.183.190:46301
I20260806 02:48:13.613544 22783 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:13.614076 22783 heartbeater.cc:511] Master 127.1.183.190:46301 requested a full tablet report, sending...
I20260806 02:48:13.615237 22309 ts_manager.cc:194] Registered new tserver with Master: 5d733086cd7249c08b869c31d708b865 (127.1.183.130:37461)
I20260806 02:48:13.615868 22309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.130@KRBTEST.COM'} at 127.1.183.130:42263
WARNING: no policy specified for kudu/127.1.183.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
I20260806 02:48:13.639326 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.131:0
--local_ip_for_outbound_sockets=127.1.183.131
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:46301
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:38541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:48:13.759202 22793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:13.759550 22793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:13.759632 22793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:13.764002 22793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:13.764106 22793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:13.764256 22793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.131
I20260806 02:48:13.769757 22793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:38541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:46301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.131
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:48:13.771008 22793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:13.772115 22793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:13.782660 22806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:13.782646 22802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:13.782943 22793 server_base.cc:1034] running on GCE node
W20260806 02:48:13.782630 22800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:13.783475 22793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:13.784266 22793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:13.785468 22793 hybrid_clock.cc:648] HybridClock initialized: now 1785984493785451 us; error 33 us; skew 500 ppm
Aug 06 02:48:13 dist-test-slave-gmjp krb5kdc[22240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984493, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:13.788483 22793 init.cc:377] Logged in from keytab as kudu/127.1.183.131@KRBTEST.COM (short username kudu)
I20260806 02:48:13.789467 22793 webserver.cc:460] Webserver started at http://127.1.183.131:39195/ using document root <none> and password file <none>
I20260806 02:48:13.790076 22793 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:13.790171 22793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:13.790422 22793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:48:13.792790 22793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/data/instance:
uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a"
format_stamp: "Formatted at 2026-08-06 02:48:13 on dist-test-slave-gmjp"
server_key: "1e67b15636ba06b43f77b5e22e9b7307"
server_key_iv: "4f158e78b3ae16f15217bc0ab35947bb"
server_key_version: "encryptionkey@0"
I20260806 02:48:13.793388 22793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance:
uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a"
format_stamp: "Formatted at 2026-08-06 02:48:13 on dist-test-slave-gmjp"
server_key: "1e67b15636ba06b43f77b5e22e9b7307"
server_key_iv: "4f158e78b3ae16f15217bc0ab35947bb"
server_key_version: "encryptionkey@0"
I20260806 02:48:13.797425 22793 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260806 02:48:13.799973 22819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:13.801324 22793 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 02:48:13.801472 22793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/wal
uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a"
format_stamp: "Formatted at 2026-08-06 02:48:13 on dist-test-slave-gmjp"
server_key: "1e67b15636ba06b43f77b5e22e9b7307"
server_key_iv: "4f158e78b3ae16f15217bc0ab35947bb"
server_key_version: "encryptionkey@0"
I20260806 02:48:13.801595 22793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:13.821208 22793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:13.824374 22793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:13.824587 22793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:13.825224 22793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:48:13.826110 22793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:48:13.826184 22793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:13.826259 22793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:48:13.826313 22793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:13.835690 22793 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.131:45667
I20260806 02:48:13.835709 22987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.131:45667 every 8 connection(s)
I20260806 02:48:13.836670 22793 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
I20260806 02:48:13.837659 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 22793
I20260806 02:48:13.837785 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance
I20260806 02:48:13.838100 1758 external_mini_cluster.cc:1428] Setting key 344d9b7c1c902c9e155d9fc804b1592d
Aug 06 02:48:13 dist-test-slave-gmjp krb5kdc[22240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984493, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:48:13.851626 22365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:13.838783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:45353 (local address 127.1.183.190:46301)
0806 02:48:13.838949 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:13.838953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:13.840061 (+ 1108us) server_negotiation.cc:407] Connection header received
0806 02:48:13.841526 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:13.841529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:13.841585 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:13.841661 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:13.842843 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:13.843392 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:13.844784 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:13.845039 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:13.845161 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:13.847692 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:13.847696 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:13.847699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:13.847759 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:13.849166 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:13.849736 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:13.849739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:13.849741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:13.849829 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:13.850202 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:13.850207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:13.850209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:13.850410 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:13.850521 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:13.851291 (+ 770us) server_negotiation.cc:299] Negotiation successful
0806 02:48:13.851444 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 02:48:13.852555 22991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:13.839293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.131:45353)
0806 02:48:13.839920 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:13.839965 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:13.841318 (+ 1353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:13.841843 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:13.841849 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:13.842205 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:13.842687 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:13.842700 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:13.843525 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:13.843528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:13.844541 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:13.844551 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:13.845219 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:13.845223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:13.845401 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:13.845961 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:13.845978 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:13.847507 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:13.849312 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:13.849318 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:13.849335 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:13.849619 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:13.849936 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:13.849940 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:13.849942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:13.850101 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:13.850519 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:13.850527 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:13.850846 (+ 319us) client_negotiation.cc:769] Sending connection context
0806 02:48:13.851073 (+ 227us) client_negotiation.cc:240] Negotiation successful
0806 02:48:13.851458 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":172,"threads_started":1}
I20260806 02:48:13.853650 22988 heartbeater.cc:347] Connected to a master server at 127.1.183.190:46301
I20260806 02:48:13.853940 22988 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:13.854482 22988 heartbeater.cc:511] Master 127.1.183.190:46301 requested a full tablet report, sending...
I20260806 02:48:13.855538 22309 ts_manager.cc:194] Registered new tserver with Master: 48b3dfeef0cd4ba4b711d09bc2410f4a (127.1.183.131:45667)
I20260806 02:48:13.856258 22309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.131@KRBTEST.COM'} at 127.1.183.131:45353
I20260806 02:48:13.864856 1758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:48:13.874818 22365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:13.866922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40542 (local address 127.1.183.190:46301)
0806 02:48:13.867100 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:13.867107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:13.867295 (+ 188us) server_negotiation.cc:407] Connection header received
0806 02:48:13.867482 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:13.867485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:13.867530 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:13.867614 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:13.868427 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:13.868961 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:13.869842 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:13.870071 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:13.870188 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:13.871589 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:13.871595 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:13.871603 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:13.871627 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:13.872871 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:13.873331 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:13.873340 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:13.873341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:13.873416 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:13.873696 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:13.873700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:13.873701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:13.874133 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:13.874272 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:13.874526 (+ 254us) server_negotiation.cc:299] Negotiation successful
0806 02:48:13.874647 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:48:13.877868 22309 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:48:13.881032 22309 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:48:13.895475 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:13.889640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41170 (local address 127.1.183.129:40057)
0806 02:48:13.890018 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:13.890026 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:13.890258 (+ 232us) server_negotiation.cc:407] Connection header received
0806 02:48:13.890353 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:13.890358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:13.890568 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:13.890726 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:13.891246 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:13.892429 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:13.894118 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:13.894793 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:13.894920 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:13.895088 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:13.895223 (+ 135us) server_negotiation.cc:299] Negotiation successful
0806 02:48:13.895318 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":64,"threads_started":1}
I20260806 02:48:13.895466 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:13.889453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:40057 (local address 127.0.0.1:41170)
0806 02:48:13.889968 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:13.890021 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:13.890163 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:13.890922 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:13.890927 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:13.890948 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:13.891117 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:13.891125 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:13.892552 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:13.892555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:13.893993 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:13.894004 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:13.894923 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:13.894927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:13.895031 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:13.895067 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 02:48:13.895175 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 02:48:13.895312 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":1,"thread_start_us":111,"threads_started":1}
I20260806 02:48:13.896023 23011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:13.890385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41584 (local address 127.1.183.131:45667)
0806 02:48:13.890823 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:13.890834 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:13.891086 (+ 252us) server_negotiation.cc:407] Connection header received
0806 02:48:13.891320 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:13.891326 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:13.891550 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:13.891695 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:13.891913 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:13.892921 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:13.894449 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:13.895227 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:13.895316 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:13.895456 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:13.895766 (+ 310us) server_negotiation.cc:299] Negotiation successful
0806 02:48:13.895864 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":125,"threads_started":1}
I20260806 02:48:13.896286 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:13.890380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:37461 (local address 127.0.0.1:58132)
0806 02:48:13.891332 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:13.891344 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:13.891430 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:13.891939 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:13.891942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:13.891963 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:13.892043 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:13.892049 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:13.893259 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:13.893262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:13.894245 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:13.894256 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:13.895526 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:13.895530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:13.895926 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:13.895950 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 02:48:13.896035 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 02:48:13.896123 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"thread_start_us":105,"threads_started":1}
I20260806 02:48:13.896365 23012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:13.890813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58132 (local address 127.1.183.130:37461)
0806 02:48:13.891181 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:13.891196 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:13.891446 (+ 250us) server_negotiation.cc:407] Connection header received
0806 02:48:13.891554 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:13.891559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:13.891735 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:13.891872 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:13.892182 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:13.893113 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:13.894401 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:13.895227 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:13.895330 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:13.895456 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:13.896081 (+ 625us) server_negotiation.cc:299] Negotiation successful
0806 02:48:13.896227 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":135,"threads_started":1}
I20260806 02:48:13.897210 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:13.889734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:45667 (local address 127.0.0.1:41584)
0806 02:48:13.890995 (+ 1261us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:13.891009 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:13.891157 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:13.891683 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:13.891685 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:13.891703 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:13.891799 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:13.891805 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:13.893081 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:13.893084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:13.894273 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:13.894281 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:13.895545 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:13.895548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:13.895622 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:13.895643 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 02:48:13.896978 (+ 1335us) client_negotiation.cc:240] Negotiation successful
0806 02:48:13.897060 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1178,"mutex_wait_us":106,"thread_start_us":107,"threads_started":1}
I20260806 02:48:13.898752 22487 tablet_service.cc:1467] Processing CreateTablet for tablet f4064e2f85e7476984cd753b1dab7d56 (DEFAULT_TABLE table=test-table [id=2588ba717a3a4fb4a051fdf9dc354afc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:48:13.898819 22689 tablet_service.cc:1467] Processing CreateTablet for tablet f4064e2f85e7476984cd753b1dab7d56 (DEFAULT_TABLE table=test-table [id=2588ba717a3a4fb4a051fdf9dc354afc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:48:13.899753 22689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4064e2f85e7476984cd753b1dab7d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:48:13.899755 22487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4064e2f85e7476984cd753b1dab7d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:48:13.902956 22889 tablet_service.cc:1467] Processing CreateTablet for tablet f4064e2f85e7476984cd753b1dab7d56 (DEFAULT_TABLE table=test-table [id=2588ba717a3a4fb4a051fdf9dc354afc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:48:13.904114 22889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4064e2f85e7476984cd753b1dab7d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:48:13.906360 23013 tablet_bootstrap.cc:492] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977: Bootstrap starting.
I20260806 02:48:13.908557 23014 tablet_bootstrap.cc:492] T f4064e2f85e7476984cd753b1dab7d56 P 5d733086cd7249c08b869c31d708b865: Bootstrap starting.
I20260806 02:48:13.908905 23013 tablet_bootstrap.cc:654] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:13.909674 23013 log.cc:824] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:13.911437 23014 tablet_bootstrap.cc:654] T f4064e2f85e7476984cd753b1dab7d56 P 5d733086cd7249c08b869c31d708b865: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:13.912354 23013 tablet_bootstrap.cc:492] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977: No bootstrap required, opened a new log
I20260806 02:48:13.912395 23014 log.cc:824] T f4064e2f85e7476984cd753b1dab7d56 P 5d733086cd7249c08b869c31d708b865: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:13.912568 23013 ts_tablet_manager.cc:1397] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:48:13.912562 23017 tablet_bootstrap.cc:492] T f4064e2f85e7476984cd753b1dab7d56 P 48b3dfeef0cd4ba4b711d09bc2410f4a: Bootstrap starting.
I20260806 02:48:13.914562 23014 tablet_bootstrap.cc:492] T f4064e2f85e7476984cd753b1dab7d56 P 5d733086cd7249c08b869c31d708b865: No bootstrap required, opened a new log
I20260806 02:48:13.914774 23014 ts_tablet_manager.cc:1397] T f4064e2f85e7476984cd753b1dab7d56 P 5d733086cd7249c08b869c31d708b865: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260806 02:48:13.914798 23017 tablet_bootstrap.cc:654] T f4064e2f85e7476984cd753b1dab7d56 P 48b3dfeef0cd4ba4b711d09bc2410f4a: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:13.916044 23017 log.cc:824] T f4064e2f85e7476984cd753b1dab7d56 P 48b3dfeef0cd4ba4b711d09bc2410f4a: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:13.916348 23013 raft_consensus.cc:348] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe3e718f749433d879f26d19ec8e977" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40057 } } peers { permanent_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45667 } } peers { permanent_uuid: "5d733086cd7249c08b869c31d708b865" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37461 } }
I20260806 02:48:13.916657 23013 raft_consensus.cc:374] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:13.916729 23013 raft_consensus.cc:729] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebe3e718f749433d879f26d19ec8e977, State: Initialized, Role: FOLLOWER
I20260806 02:48:13.917230 23013 consensus_queue.cc:260] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe3e718f749433d879f26d19ec8e977" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40057 } } peers { permanent_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45667 } } peers { permanent_uuid: "5d733086cd7249c08b869c31d708b865" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37461 } }
I20260806 02:48:13.917950 22580 heartbeater.cc:503] Master 127.1.183.190:46301 was elected leader, sending a full tablet report...
I20260806 02:48:13.918329 23017 tablet_bootstrap.cc:492] T f4064e2f85e7476984cd753b1dab7d56 P 48b3dfeef0cd4ba4b711d09bc2410f4a: No bootstrap required, opened a new log
I20260806 02:48:13.918550 23017 ts_tablet_manager.cc:1397] T f4064e2f85e7476984cd753b1dab7d56 P 48b3dfeef0cd4ba4b711d09bc2410f4a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:48:13.918334 23013 ts_tablet_manager.cc:1428] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:48:13.919934 23014 raft_consensus.cc:348] T f4064e2f85e7476984cd753b1dab7d56 P 5d733086cd7249c08b869c31d708b865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe3e718f749433d879f26d19ec8e977" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40057 } } peers { permanent_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45667 } } peers { permanent_uuid: "5d733086cd7249c08b869c31d708b865" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37461 } }
I20260806 02:48:13.920220 23014 raft_consensus.cc:374] T f4064e2f85e7476984cd753b1dab7d56 P 5d733086cd7249c08b869c31d708b865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:13.920284 23014 raft_consensus.cc:729] T f4064e2f85e7476984cd753b1dab7d56 P 5d733086cd7249c08b869c31d708b865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d733086cd7249c08b869c31d708b865, State: Initialized, Role: FOLLOWER
I20260806 02:48:13.920744 23014 consensus_queue.cc:260] T f4064e2f85e7476984cd753b1dab7d56 P 5d733086cd7249c08b869c31d708b865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe3e718f749433d879f26d19ec8e977" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40057 } } peers { permanent_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45667 } } peers { permanent_uuid: "5d733086cd7249c08b869c31d708b865" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37461 } }
I20260806 02:48:13.921442 22783 heartbeater.cc:503] Master 127.1.183.190:46301 was elected leader, sending a full tablet report...
I20260806 02:48:13.921777 23014 ts_tablet_manager.cc:1428] T f4064e2f85e7476984cd753b1dab7d56 P 5d733086cd7249c08b869c31d708b865: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260806 02:48:13.923110 23017 raft_consensus.cc:348] T f4064e2f85e7476984cd753b1dab7d56 P 48b3dfeef0cd4ba4b711d09bc2410f4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe3e718f749433d879f26d19ec8e977" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40057 } } peers { permanent_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45667 } } peers { permanent_uuid: "5d733086cd7249c08b869c31d708b865" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37461 } }
I20260806 02:48:13.923383 23017 raft_consensus.cc:374] T f4064e2f85e7476984cd753b1dab7d56 P 48b3dfeef0cd4ba4b711d09bc2410f4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:13.923441 23017 raft_consensus.cc:729] T f4064e2f85e7476984cd753b1dab7d56 P 48b3dfeef0cd4ba4b711d09bc2410f4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48b3dfeef0cd4ba4b711d09bc2410f4a, State: Initialized, Role: FOLLOWER
I20260806 02:48:13.923875 23017 consensus_queue.cc:260] T f4064e2f85e7476984cd753b1dab7d56 P 48b3dfeef0cd4ba4b711d09bc2410f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe3e718f749433d879f26d19ec8e977" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40057 } } peers { permanent_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45667 } } peers { permanent_uuid: "5d733086cd7249c08b869c31d708b865" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37461 } }
I20260806 02:48:13.924562 22988 heartbeater.cc:503] Master 127.1.183.190:46301 was elected leader, sending a full tablet report...
I20260806 02:48:13.924885 23017 ts_tablet_manager.cc:1428] T f4064e2f85e7476984cd753b1dab7d56 P 48b3dfeef0cd4ba4b711d09bc2410f4a: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
W20260806 02:48:14.087822 22990 tablet.cc:2369] T f4064e2f85e7476984cd753b1dab7d56 P 48b3dfeef0cd4ba4b711d09bc2410f4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:48:14.096774 22784 tablet.cc:2369] T f4064e2f85e7476984cd753b1dab7d56 P 5d733086cd7249c08b869c31d708b865: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:48:14.106423 22583 tablet.cc:2369] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:48:14.168402 23021 raft_consensus.cc:482] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:48:14.168632 23021 raft_consensus.cc:504] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe3e718f749433d879f26d19ec8e977" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40057 } } peers { permanent_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45667 } } peers { permanent_uuid: "5d733086cd7249c08b869c31d708b865" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37461 } }
I20260806 02:48:14.169785 23021 leader_election.cc:288] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48b3dfeef0cd4ba4b711d09bc2410f4a (127.1.183.131:45667), 5d733086cd7249c08b869c31d708b865 (127.1.183.130:37461)
I20260806 02:48:14.174172 23012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:14.170365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:40869 (local address 127.1.183.130:37461)
0806 02:48:14.170517 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:14.170525 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:14.170698 (+ 173us) server_negotiation.cc:407] Connection header received
0806 02:48:14.170924 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:14.170932 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:14.170977 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:14.171067 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:14.171369 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:14.171857 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:14.173121 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:14.173589 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:14.173718 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:14.173775 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:14.173961 (+ 186us) server_negotiation.cc:299] Negotiation successful
0806 02:48:14.174032 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:48:14.174232 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:14.170120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:37461 (local address 127.1.183.129:40869)
0806 02:48:14.170588 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:14.170604 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:14.170782 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:14.171077 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:14.171080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:14.171119 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:14.171241 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:14.171246 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:14.171973 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:14.171978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:14.172993 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:14.173003 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:14.173713 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:14.173717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:14.173816 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:14.173848 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 02:48:14.173943 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 02:48:14.174021 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":126,"spinlock_wait_cycles":1280,"thread_start_us":128,"threads_started":1}
I20260806 02:48:14.174221 23011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:14.170627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:53683 (local address 127.1.183.131:45667)
0806 02:48:14.170804 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:14.170807 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:14.170819 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:48:14.170868 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:14.170871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:14.170908 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:14.171007 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:14.171332 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:14.171847 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:14.173145 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:14.173638 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:14.173774 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:14.173826 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:14.174010 (+ 184us) server_negotiation.cc:299] Negotiation successful
0806 02:48:14.174098 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 02:48:14.174242 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:14.170120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:45667 (local address 127.1.183.129:53683)
0806 02:48:14.170515 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:14.170531 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:14.170651 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:14.171032 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:14.171035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:14.171052 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:14.171213 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:14.171218 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:14.171971 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:14.171977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:14.173014 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:14.173025 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:14.173764 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:14.173768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:14.173870 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:14.173892 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:48:14.173992 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 02:48:14.174067 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":24,"spinlock_wait_cycles":4224,"thread_start_us":133,"threads_started":1}
I20260806 02:48:14.175218 22919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4064e2f85e7476984cd753b1dab7d56" candidate_uuid: "ebe3e718f749433d879f26d19ec8e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" is_pre_election: true
I20260806 02:48:14.175305 22720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4064e2f85e7476984cd753b1dab7d56" candidate_uuid: "ebe3e718f749433d879f26d19ec8e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d733086cd7249c08b869c31d708b865" is_pre_election: true
I20260806 02:48:14.175590 22919 raft_consensus.cc:2454] T f4064e2f85e7476984cd753b1dab7d56 P 48b3dfeef0cd4ba4b711d09bc2410f4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebe3e718f749433d879f26d19ec8e977 in term 0.
I20260806 02:48:14.175590 22720 raft_consensus.cc:2454] T f4064e2f85e7476984cd753b1dab7d56 P 5d733086cd7249c08b869c31d708b865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebe3e718f749433d879f26d19ec8e977 in term 0.
I20260806 02:48:14.176090 22419 leader_election.cc:302] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48b3dfeef0cd4ba4b711d09bc2410f4a, ebe3e718f749433d879f26d19ec8e977; no voters:
I20260806 02:48:14.176450 23021 raft_consensus.cc:2781] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:48:14.176544 23021 raft_consensus.cc:482] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:48:14.176587 23021 raft_consensus.cc:3037] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:14.177675 23021 raft_consensus.cc:504] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe3e718f749433d879f26d19ec8e977" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40057 } } peers { permanent_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45667 } } peers { permanent_uuid: "5d733086cd7249c08b869c31d708b865" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37461 } }
I20260806 02:48:14.178058 23021 leader_election.cc:288] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977 [CANDIDATE]: Term 1 election: Requested vote from peers 48b3dfeef0cd4ba4b711d09bc2410f4a (127.1.183.131:45667), 5d733086cd7249c08b869c31d708b865 (127.1.183.130:37461)
I20260806 02:48:14.178416 22919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4064e2f85e7476984cd753b1dab7d56" candidate_uuid: "ebe3e718f749433d879f26d19ec8e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a"
I20260806 02:48:14.178491 22720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4064e2f85e7476984cd753b1dab7d56" candidate_uuid: "ebe3e718f749433d879f26d19ec8e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d733086cd7249c08b869c31d708b865"
I20260806 02:48:14.178572 22919 raft_consensus.cc:3037] T f4064e2f85e7476984cd753b1dab7d56 P 48b3dfeef0cd4ba4b711d09bc2410f4a [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:14.178634 22720 raft_consensus.cc:3037] T f4064e2f85e7476984cd753b1dab7d56 P 5d733086cd7249c08b869c31d708b865 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:14.179947 22720 raft_consensus.cc:2454] T f4064e2f85e7476984cd753b1dab7d56 P 5d733086cd7249c08b869c31d708b865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebe3e718f749433d879f26d19ec8e977 in term 1.
I20260806 02:48:14.179947 22919 raft_consensus.cc:2454] T f4064e2f85e7476984cd753b1dab7d56 P 48b3dfeef0cd4ba4b711d09bc2410f4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebe3e718f749433d879f26d19ec8e977 in term 1.
I20260806 02:48:14.180290 22419 leader_election.cc:302] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48b3dfeef0cd4ba4b711d09bc2410f4a, ebe3e718f749433d879f26d19ec8e977; no voters:
I20260806 02:48:14.180454 23021 raft_consensus.cc:2781] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:48:14.182232 23021 raft_consensus.cc:686] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977 [term 1 LEADER]: Becoming Leader. State: Replica: ebe3e718f749433d879f26d19ec8e977, State: Running, Role: LEADER
I20260806 02:48:14.182626 23021 consensus_queue.cc:237] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe3e718f749433d879f26d19ec8e977" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40057 } } peers { permanent_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45667 } } peers { permanent_uuid: "5d733086cd7249c08b869c31d708b865" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37461 } }
I20260806 02:48:14.186262 22309 catalog_manager.cc:5495] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977 reported cstate change: term changed from 0 to 1, leader changed from <none> to ebe3e718f749433d879f26d19ec8e977 (127.1.183.129). New cstate: current_term: 1 leader_uuid: "ebe3e718f749433d879f26d19ec8e977" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe3e718f749433d879f26d19ec8e977" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d733086cd7249c08b869c31d708b865" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37461 } health_report { overall_health: UNKNOWN } } }
I20260806 02:48:14.263824 23003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:14.259957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41176 (local address 127.1.183.129:40057)
0806 02:48:14.260132 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:14.260139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:14.260196 (+ 57us) server_negotiation.cc:407] Connection header received
0806 02:48:14.260276 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:14.260278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:14.260334 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:14.260455 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:48:14.260736 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:14.261266 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:14.262259 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:14.262486 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:14.262600 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:14.262952 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:48:14.263392 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:48:14.263535 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:14.263598 (+ 63us) server_negotiation.cc:299] Negotiation successful
0806 02:48:14.263663 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 02:48:14.270228 22919 raft_consensus.cc:1261] T f4064e2f85e7476984cd753b1dab7d56 P 48b3dfeef0cd4ba4b711d09bc2410f4a [term 1 FOLLOWER]: Refusing update from remote peer ebe3e718f749433d879f26d19ec8e977: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:48:14.270239 22720 raft_consensus.cc:1261] T f4064e2f85e7476984cd753b1dab7d56 P 5d733086cd7249c08b869c31d708b865 [term 1 FOLLOWER]: Refusing update from remote peer ebe3e718f749433d879f26d19ec8e977: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:48:14.271023 23021 consensus_queue.cc:1035] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d733086cd7249c08b869c31d708b865" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:48:14.271273 23030 consensus_queue.cc:1035] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:48:14.278726 23033 mvcc.cc:205] Tried to move back new op lower bound from 7315392488522526720 to 7315392488175173632. Current Snapshot: MvccSnapshot[applied={T|T < 7315392488522526720}]
I20260806 02:48:14.281736 23037 mvcc.cc:205] Tried to move back new op lower bound from 7315392488522526720 to 7315392488175173632. Current Snapshot: MvccSnapshot[applied={T|T < 7315392488522526720}]
I20260806 02:48:14.290128 22306 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:48:14.290351 22306 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:48:14.293705 22306 catalog_manager.cc:5782] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2: Sending DeleteTablet for 3 replicas of tablet f4064e2f85e7476984cd753b1dab7d56
I20260806 02:48:14.294750 22889 tablet_service.cc:1514] Processing DeleteTablet for tablet f4064e2f85e7476984cd753b1dab7d56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:48:14 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:41584
I20260806 02:48:14.294734 22487 tablet_service.cc:1514] Processing DeleteTablet for tablet f4064e2f85e7476984cd753b1dab7d56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:48:14 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:41170
I20260806 02:48:14.294855 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 22395
I20260806 02:48:14.299829 23046 tablet_replica.cc:330] T f4064e2f85e7476984cd753b1dab7d56 P 48b3dfeef0cd4ba4b711d09bc2410f4a: stopping tablet replica
I20260806 02:48:14.299742 22689 tablet_service.cc:1514] Processing DeleteTablet for tablet f4064e2f85e7476984cd753b1dab7d56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:48:14 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:58132
I20260806 02:48:14.300426 23046 raft_consensus.cc:2229] T f4064e2f85e7476984cd753b1dab7d56 P 48b3dfeef0cd4ba4b711d09bc2410f4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:48:14.300838 23046 raft_consensus.cc:2258] T f4064e2f85e7476984cd753b1dab7d56 P 48b3dfeef0cd4ba4b711d09bc2410f4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:48:14.301580 23050 tablet_replica.cc:330] T f4064e2f85e7476984cd753b1dab7d56 P 5d733086cd7249c08b869c31d708b865: stopping tablet replica
I20260806 02:48:14.301936 23050 raft_consensus.cc:2229] T f4064e2f85e7476984cd753b1dab7d56 P 5d733086cd7249c08b869c31d708b865 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:48:14.302275 23050 raft_consensus.cc:2258] T f4064e2f85e7476984cd753b1dab7d56 P 5d733086cd7249c08b869c31d708b865 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:48:14.306394 23046 ts_tablet_manager.cc:1905] T f4064e2f85e7476984cd753b1dab7d56 P 48b3dfeef0cd4ba4b711d09bc2410f4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:48:14.306536 23050 ts_tablet_manager.cc:1905] T f4064e2f85e7476984cd753b1dab7d56 P 5d733086cd7249c08b869c31d708b865: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:48:14.309337 23046 ts_tablet_manager.cc:1918] T f4064e2f85e7476984cd753b1dab7d56 P 48b3dfeef0cd4ba4b711d09bc2410f4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:48:14.309491 23046 log.cc:1196] T f4064e2f85e7476984cd753b1dab7d56 P 48b3dfeef0cd4ba4b711d09bc2410f4a: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/wal/wals/f4064e2f85e7476984cd753b1dab7d56
I20260806 02:48:14.309821 23046 ts_tablet_manager.cc:1939] T f4064e2f85e7476984cd753b1dab7d56 P 48b3dfeef0cd4ba4b711d09bc2410f4a: Deleting consensus metadata
I20260806 02:48:14.310258 23050 ts_tablet_manager.cc:1918] T f4064e2f85e7476984cd753b1dab7d56 P 5d733086cd7249c08b869c31d708b865: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:48:14.310389 23050 log.cc:1196] T f4064e2f85e7476984cd753b1dab7d56 P 5d733086cd7249c08b869c31d708b865: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/wal/wals/f4064e2f85e7476984cd753b1dab7d56
I20260806 02:48:14.310726 23050 ts_tablet_manager.cc:1939] T f4064e2f85e7476984cd753b1dab7d56 P 5d733086cd7249c08b869c31d708b865: Deleting consensus metadata
I20260806 02:48:14.311415 22287 catalog_manager.cc:4841] TS 48b3dfeef0cd4ba4b711d09bc2410f4a (127.1.183.131:45667): tablet f4064e2f85e7476984cd753b1dab7d56 (table test-table [id=2588ba717a3a4fb4a051fdf9dc354afc]) successfully deleted
I20260806 02:48:14.312055 22290 catalog_manager.cc:4841] TS 5d733086cd7249c08b869c31d708b865 (127.1.183.130:37461): tablet f4064e2f85e7476984cd753b1dab7d56 (table test-table [id=2588ba717a3a4fb4a051fdf9dc354afc]) successfully deleted
I20260806 02:48:14.314934 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 22589
W20260806 02:48:14.314934 22288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.183.129:40057 (error 108)
I20260806 02:48:14.323745 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:14.323388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:40057 (local address 127.0.0.1:41180)
0806 02:48:14.323550 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:14.323644 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.129:40057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260806 02:48:14.323961 22288 catalog_manager.cc:4584] TS ebe3e718f749433d879f26d19ec8e977 (127.1.183.129:40057): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f4064e2f85e7476984cd753b1dab7d56: Network error: Client connection negotiation failed: client connection to 127.1.183.129:40057: connect: Connection refused (error 111)
I20260806 02:48:14.328577 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 22793
I20260806 02:48:14.347858 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 22262
I20260806 02:48:14.359241 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:46301
--webserver_interface=127.1.183.190
--webserver_port=43579
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:38541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:46301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:48:14.479586 23053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:14.480002 23053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:14.480116 23053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:14.484398 23053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:48:14.484519 23053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:14.484558 23053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:14.484598 23053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:48:14.484634 23053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:48:14.490237 23053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:38541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:46301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:46301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=43579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:48:14.491431 23053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:14.492560 23053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:14.506780 23064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:14.506799 23061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:14.506780 23062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:14.507436 23053 server_base.cc:1034] running on GCE node
I20260806 02:48:14.508116 23053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:14.509486 23053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:14.510679 23053 hybrid_clock.cc:648] HybridClock initialized: now 1785984494510637 us; error 51 us; skew 500 ppm
Aug 06 02:48:14 dist-test-slave-gmjp krb5kdc[22240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984494, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:14.513752 23053 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:48:14.514820 23053 webserver.cc:460] Webserver started at http://127.1.183.190:43579/ using document root <none> and password file <none>
I20260806 02:48:14.515462 23053 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:14.515588 23053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:14.519976 23053 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:48:14.522318 23075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:14.523659 23053 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:48:14.523869 23053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "efc9d63000364ad9a38827aededd20e2"
format_stamp: "Formatted at 2026-08-06 02:48:12 on dist-test-slave-gmjp"
server_key: "4526a6d2f8b50564bde8aae1fcdf5fb7"
server_key_iv: "9b3c512375fd98816072daaee70c5ec7"
server_key_version: "encryptionkey@0"
I20260806 02:48:14.524618 23053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:14.554217 23053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:14.557924 23053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:14.558162 23053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:14.566051 23053 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:46301
I20260806 02:48:14.566383 23158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:46301 every 8 connection(s)
I20260806 02:48:14.567286 23053 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:48:14.567852 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 23053
I20260806 02:48:14.568398 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.129:40057
--local_ip_for_outbound_sockets=127.1.183.129
--tserver_master_addrs=127.1.183.190:46301
--webserver_port=40849
--webserver_interface=127.1.183.129
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:38541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:48:14.572378 23159 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:48:14.575240 23159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2: Bootstrap starting.
I20260806 02:48:14.596083 23159 log.cc:824] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:14.606050 23159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:48:14.606509 23159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2: Bootstrap complete.
I20260806 02:48:14.610832 23159 raft_consensus.cc:348] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc9d63000364ad9a38827aededd20e2" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 46301 } }
I20260806 02:48:14.611294 23159 raft_consensus.cc:729] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: efc9d63000364ad9a38827aededd20e2, State: Initialized, Role: FOLLOWER
I20260806 02:48:14.611804 23159 consensus_queue.cc:260] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc9d63000364ad9a38827aededd20e2" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 46301 } }
I20260806 02:48:14.611930 23159 raft_consensus.cc:388] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:48:14.612032 23159 raft_consensus.cc:482] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:48:14.612156 23159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:48:14.614328 23159 raft_consensus.cc:504] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc9d63000364ad9a38827aededd20e2" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 46301 } }
I20260806 02:48:14.614687 23159 leader_election.cc:302] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efc9d63000364ad9a38827aededd20e2; no voters:
I20260806 02:48:14.615067 23159 leader_election.cc:288] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 02:48:14.615161 23164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 02:48:14.615377 23164 raft_consensus.cc:686] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [term 2 LEADER]: Becoming Leader. State: Replica: efc9d63000364ad9a38827aededd20e2, State: Running, Role: LEADER
I20260806 02:48:14.615679 23164 consensus_queue.cc:237] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc9d63000364ad9a38827aededd20e2" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 46301 } }
I20260806 02:48:14.616446 23159 sys_catalog.cc:564] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:48:14.617298 23167 sys_catalog.cc:455] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efc9d63000364ad9a38827aededd20e2. Latest consensus state: current_term: 2 leader_uuid: "efc9d63000364ad9a38827aededd20e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc9d63000364ad9a38827aededd20e2" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 46301 } } }
I20260806 02:48:14.617427 23167 sys_catalog.cc:458] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [sys.catalog]: This master's current role is: LEADER
I20260806 02:48:14.617354 23165 sys_catalog.cc:455] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "efc9d63000364ad9a38827aededd20e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc9d63000364ad9a38827aededd20e2" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 46301 } } }
I20260806 02:48:14.617486 23165 sys_catalog.cc:458] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [sys.catalog]: This master's current role is: LEADER
I20260806 02:48:14.621465 23186 catalog_manager.cc:1259] Loaded cluster ID: 7ca99d4d402d42278037be43e9849523
I20260806 02:48:14.621524 23186 catalog_manager.cc:1552] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2: loading cluster ID for follower catalog manager: success
W20260806 02:48:14.622313 23186 catalog_manager.cc:1576] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260806 02:48:14.622382 23186 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260806 02:48:14.622555 23174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:48:14.624727 23174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:48:14.625164 23174 catalog_manager.cc:1259] Loaded cluster ID: 7ca99d4d402d42278037be43e9849523
I20260806 02:48:14.625236 23174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:48:14.626031 23174 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260806 02:48:14.629063 23174 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260806 02:48:14.633952 23174 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:48:14.634692 23174 catalog_manager.cc:5879] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2: Loaded TSK: 0
I20260806 02:48:14.635658 23174 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:48:14.704916 23162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:14.705191 23162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:14.705237 23162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:14.709481 23162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:14.709558 23162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:14.709649 23162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.129
I20260806 02:48:14.715247 23162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:38541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.129:40057
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.183.129
--webserver_port=40849
--tserver_master_addrs=127.1.183.190:46301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.129
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:48:14.716552 23162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:14.717841 23162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:14.727758 23196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:14.727758 23198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:14.727901 23201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:14.728127 23162 server_base.cc:1034] running on GCE node
I20260806 02:48:14.728590 23162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:14.729348 23162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:14.730515 23162 hybrid_clock.cc:648] HybridClock initialized: now 1785984494730489 us; error 41 us; skew 500 ppm
Aug 06 02:48:14 dist-test-slave-gmjp krb5kdc[22240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984494, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:14.733438 23162 init.cc:377] Logged in from keytab as kudu/127.1.183.129@KRBTEST.COM (short username kudu)
I20260806 02:48:14.734493 23162 webserver.cc:460] Webserver started at http://127.1.183.129:40849/ using document root <none> and password file <none>
I20260806 02:48:14.735074 23162 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:14.735153 23162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:14.739275 23162 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:48:14.741648 23209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:14.743009 23162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:48:14.743110 23162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/wal
uuid: "ebe3e718f749433d879f26d19ec8e977"
format_stamp: "Formatted at 2026-08-06 02:48:13 on dist-test-slave-gmjp"
server_key: "dc2ec2dde1a60ae694d399cc8137925d"
server_key_iv: "35ab59c11508b49a521ad83ac33ad786"
server_key_version: "encryptionkey@0"
I20260806 02:48:14.743693 23162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:14.761478 23162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:14.765208 23162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:14.765443 23162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:14.766218 23162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:48:14.767550 23220 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 02:48:14.771855 23162 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 02:48:14.771934 23162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260806 02:48:14.772012 23162 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 02:48:14.773846 23162 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 02:48:14.773912 23162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 02:48:14.774191 23220 tablet_bootstrap.cc:492] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977: Bootstrap starting.
I20260806 02:48:14.785751 23162 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.129:40057
I20260806 02:48:14.785801 23389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.129:40057 every 8 connection(s)
I20260806 02:48:14.787174 23162 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
I20260806 02:48:14.787490 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 23162
I20260806 02:48:14.788012 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.130:37461
--local_ip_for_outbound_sockets=127.1.183.130
--tserver_master_addrs=127.1.183.190:46301
--webserver_port=39981
--webserver_interface=127.1.183.130
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:38541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:48:14.797492 23220 log.cc:824] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977: Log is configured to *not* fsync() on all Append() calls
Aug 06 02:48:14 dist-test-slave-gmjp krb5kdc[22240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984494, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:48:14.806384 23220 tablet_bootstrap.cc:492] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:48:14.806890 23220 tablet_bootstrap.cc:492] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977: Bootstrap complete.
I20260806 02:48:14.808506 23220 ts_tablet_manager.cc:1397] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977: Time spent bootstrapping tablet: real 0.035s user 0.019s sys 0.014s
I20260806 02:48:14.808692 23395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:14.789860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.129:46795)
0806 02:48:14.790895 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:14.790962 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:14.792170 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:14.793738 (+ 1568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:14.793749 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:14.794363 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:14.795149 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:14.795175 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:14.797081 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:14.797084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:14.798303 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:14.798314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:14.799290 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:14.799296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:14.799537 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:14.800318 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:14.800347 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:14.802460 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:14.805072 (+ 2612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:14.805079 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:14.805092 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:14.805459 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:14.805859 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:14.805863 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:14.805865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:14.806036 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:14.806734 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:14.806740 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:14.807049 (+ 309us) client_negotiation.cc:769] Sending connection context
0806 02:48:14.807257 (+ 208us) client_negotiation.cc:240] Negotiation successful
0806 02:48:14.807638 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":180,"threads_started":1}
I20260806 02:48:14.808684 23393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:14.789881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:46795 (local address 127.1.183.190:46301)
0806 02:48:14.790516 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:14.790541 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:14.791135 (+ 594us) server_negotiation.cc:407] Connection header received
0806 02:48:14.792700 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:14.792717 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:14.793206 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:14.793659 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:14.795387 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:14.796877 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:14.798447 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:14.799117 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:14.799311 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:14.802695 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:14.802702 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:14.802720 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:14.802762 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:14.804932 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:14.805586 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:14.805592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:14.805598 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:14.805752 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:14.806141 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:14.806145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:14.806146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:14.806604 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:14.806737 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:14.807391 (+ 654us) server_negotiation.cc:299] Negotiation successful
0806 02:48:14.807702 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":141,"threads_started":1}
I20260806 02:48:14.811856 23390 heartbeater.cc:347] Connected to a master server at 127.1.183.190:46301
I20260806 02:48:14.811519 23220 raft_consensus.cc:348] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe3e718f749433d879f26d19ec8e977" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40057 } } peers { permanent_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45667 } } peers { permanent_uuid: "5d733086cd7249c08b869c31d708b865" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37461 } }
I20260806 02:48:14.812081 23220 raft_consensus.cc:729] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebe3e718f749433d879f26d19ec8e977, State: Initialized, Role: FOLLOWER
I20260806 02:48:14.812110 23390 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:14.812603 23220 consensus_queue.cc:260] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe3e718f749433d879f26d19ec8e977" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40057 } } peers { permanent_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45667 } } peers { permanent_uuid: "5d733086cd7249c08b869c31d708b865" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37461 } }
I20260806 02:48:14.812739 23390 heartbeater.cc:511] Master 127.1.183.190:46301 requested a full tablet report, sending...
I20260806 02:48:14.813508 23220 ts_tablet_manager.cc:1428] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260806 02:48:14.815316 23106 ts_manager.cc:194] Registered new tserver with Master: ebe3e718f749433d879f26d19ec8e977 (127.1.183.129:40057)
I20260806 02:48:14.819069 23106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.129@KRBTEST.COM'} at 127.1.183.129:46795
I20260806 02:48:14.821820 23390 heartbeater.cc:503] Master 127.1.183.190:46301 was elected leader, sending a full tablet report...
Aug 06 02:48:14 dist-test-slave-gmjp krb5kdc[22240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984494, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for kudu/127.1.183.129@KRBTEST.COM
I20260806 02:48:14.829972 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:14.818496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41194 (local address 127.1.183.129:40057)
0806 02:48:14.818974 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:14.818986 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:14.819432 (+ 446us) server_negotiation.cc:407] Connection header received
0806 02:48:14.819635 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:14.819640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:14.819849 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:14.820000 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:14.820699 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:14.821415 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:14.822422 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:14.822946 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:14.823062 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:14.825635 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:14.825644 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:14.825656 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:14.825679 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:14.827410 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:14.828158 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:14.828168 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:14.828199 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:14.828345 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:14.828672 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:14.828674 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:14.828675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:14.829037 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:14.829159 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:14.829632 (+ 473us) server_negotiation.cc:299] Negotiation successful
0806 02:48:14.829836 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":119,"threads_started":1}
I20260806 02:48:14.829977 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:14.818383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:40057 (local address 127.0.0.1:41194)
0806 02:48:14.819287 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:14.819331 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:14.819472 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:14.820045 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:14.820050 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:14.820461 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:14.820568 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:14.820574 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:14.821538 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:14.821542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:14.822297 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:14.822309 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:14.823084 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:14.823089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:14.823217 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:14.823848 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:14.823861 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:14.825468 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:14.827535 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:14.827546 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:14.827552 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:14.827916 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:14.828461 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:14.828464 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:14.828466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:14.828570 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:14.829165 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:14.829171 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:14.829330 (+ 159us) client_negotiation.cc:769] Sending connection context
0806 02:48:14.829516 (+ 186us) client_negotiation.cc:240] Negotiation successful
0806 02:48:14.829773 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":141,"threads_started":1}
I20260806 02:48:14.831166 23298 tablet_service.cc:1514] Processing DeleteTablet for tablet f4064e2f85e7476984cd753b1dab7d56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:48:14 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:41194
I20260806 02:48:14.832343 23402 tablet_replica.cc:330] stopping tablet replica
I20260806 02:48:14.832713 23402 raft_consensus.cc:2229] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:48:14.833035 23402 raft_consensus.cc:2258] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:48:14.834106 23402 ts_tablet_manager.cc:1905] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:48:14.838259 23402 ts_tablet_manager.cc:1918] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:48:14.838413 23402 log.cc:1196] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/wal/wals/f4064e2f85e7476984cd753b1dab7d56
I20260806 02:48:14.838799 23402 ts_tablet_manager.cc:1939] T f4064e2f85e7476984cd753b1dab7d56 P ebe3e718f749433d879f26d19ec8e977: Deleting consensus metadata
I20260806 02:48:14.840824 23078 catalog_manager.cc:4841] TS ebe3e718f749433d879f26d19ec8e977 (127.1.183.129:40057): tablet f4064e2f85e7476984cd753b1dab7d56 (table test-table [id=2588ba717a3a4fb4a051fdf9dc354afc]) successfully deleted
W20260806 02:48:14.917989 23392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:14.918311 23392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:14.918383 23392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:14.922631 23392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:14.922711 23392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:14.922775 23392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.130
I20260806 02:48:14.928443 23392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:38541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.130:37461
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.183.130
--webserver_port=39981
--tserver_master_addrs=127.1.183.190:46301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.130
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:48:14.929682 23392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:14.930749 23392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:14.941083 23416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:14.941121 23412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:14.941077 23411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:14.941473 23392 server_base.cc:1034] running on GCE node
I20260806 02:48:14.941934 23392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:14.942863 23392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:14.944067 23392 hybrid_clock.cc:648] HybridClock initialized: now 1785984494944033 us; error 53 us; skew 500 ppm
Aug 06 02:48:14 dist-test-slave-gmjp krb5kdc[22240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984494, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:14.947070 23392 init.cc:377] Logged in from keytab as kudu/127.1.183.130@KRBTEST.COM (short username kudu)
I20260806 02:48:14.948100 23392 webserver.cc:460] Webserver started at http://127.1.183.130:39981/ using document root <none> and password file <none>
I20260806 02:48:14.948736 23392 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:14.948858 23392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:14.953073 23392 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:48:14.955363 23424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:14.956743 23392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:48:14.956907 23392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/wal
uuid: "5d733086cd7249c08b869c31d708b865"
format_stamp: "Formatted at 2026-08-06 02:48:13 on dist-test-slave-gmjp"
server_key: "38a71d57c4f6d1046ac832ef7dabeb58"
server_key_iv: "e72337064f96990dc798ce17925891cf"
server_key_version: "encryptionkey@0"
I20260806 02:48:14.957573 23392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:14.970702 23392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:14.974051 23392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:14.974295 23392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:14.974941 23392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:48:14.975867 23392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:48:14.975941 23392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:14.976016 23392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:48:14.976069 23392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:14.985716 23392 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.130:37461
I20260806 02:48:14.985773 23585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.130:37461 every 8 connection(s)
I20260806 02:48:14.986831 23392 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
I20260806 02:48:14.987850 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 23392
I20260806 02:48:14.988433 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.131:45667
--local_ip_for_outbound_sockets=127.1.183.131
--tserver_master_addrs=127.1.183.190:46301
--webserver_port=39195
--webserver_interface=127.1.183.131
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:38541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:48:14 dist-test-slave-gmjp krb5kdc[22240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984494, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:48:15.002105 23393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:14.988896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:42203 (local address 127.1.183.190:46301)
0806 02:48:14.989034 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:14.989038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:14.990105 (+ 1067us) server_negotiation.cc:407] Connection header received
0806 02:48:14.992143 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:14.992146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:14.992225 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:14.992305 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:14.993586 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:14.994219 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:14.995383 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:14.995622 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:14.995748 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:14.998304 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:14.998307 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:14.998309 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:14.998337 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:14.999878 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:15.000486 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:15.000490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:15.000491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:15.000574 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:15.000930 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:15.000934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:15.000935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:15.001126 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:15.001236 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:15.001760 (+ 524us) server_negotiation.cc:299] Negotiation successful
0806 02:48:15.001920 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 02:48:15.002995 23589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:14.989408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.130:42203)
0806 02:48:14.989934 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:14.989979 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:14.991851 (+ 1872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:14.992518 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:14.992527 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:14.992914 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:14.993409 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:14.993423 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:14.994354 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:14.994357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:14.995248 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:14.995257 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:14.995792 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:14.995796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:14.995972 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:14.996630 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:14.996646 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:14.998120 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:14.999999 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:15.000005 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:15.000021 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:15.000378 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:15.000672 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:15.000675 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:15.000677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:15.000826 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:15.001235 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:15.001244 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:15.001510 (+ 266us) client_negotiation.cc:769] Sending connection context
0806 02:48:15.001705 (+ 195us) client_negotiation.cc:240] Negotiation successful
0806 02:48:15.002051 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":157,"threads_started":1}
I20260806 02:48:15.004298 23586 heartbeater.cc:347] Connected to a master server at 127.1.183.190:46301
I20260806 02:48:15.004554 23586 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:15.005054 23586 heartbeater.cc:511] Master 127.1.183.190:46301 requested a full tablet report, sending...
I20260806 02:48:15.006186 23106 ts_manager.cc:194] Registered new tserver with Master: 5d733086cd7249c08b869c31d708b865 (127.1.183.130:37461)
I20260806 02:48:15.006778 23106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.130@KRBTEST.COM'} at 127.1.183.130:42203
W20260806 02:48:15.110952 23590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:15.111281 23590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:15.111382 23590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:15.115623 23590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:15.115720 23590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:15.115813 23590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.131
I20260806 02:48:15.121692 23590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:38541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.131:45667
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.183.131
--webserver_port=39195
--tserver_master_addrs=127.1.183.190:46301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.131
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:48:15.122910 23590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:15.124042 23590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:15.134411 23600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:15.134420 23601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:15.134640 23590 server_base.cc:1034] running on GCE node
W20260806 02:48:15.134423 23604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:15.135056 23590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:15.135764 23590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:15.136960 23590 hybrid_clock.cc:648] HybridClock initialized: now 1785984495136941 us; error 28 us; skew 500 ppm
Aug 06 02:48:15 dist-test-slave-gmjp krb5kdc[22240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984495, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:15.139693 23590 init.cc:377] Logged in from keytab as kudu/127.1.183.131@KRBTEST.COM (short username kudu)
I20260806 02:48:15.140637 23590 webserver.cc:460] Webserver started at http://127.1.183.131:39195/ using document root <none> and password file <none>
I20260806 02:48:15.141242 23590 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:15.141369 23590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:15.145210 23590 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260806 02:48:15.147810 23613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:15.149207 23590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:48:15.149348 23590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/wal
uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a"
format_stamp: "Formatted at 2026-08-06 02:48:13 on dist-test-slave-gmjp"
server_key: "1e67b15636ba06b43f77b5e22e9b7307"
server_key_iv: "4f158e78b3ae16f15217bc0ab35947bb"
server_key_version: "encryptionkey@0"
I20260806 02:48:15.149911 23590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:15.180568 23590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:15.185222 23590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:15.185477 23590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:15.186369 23590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:48:15.187366 23590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:48:15.187418 23590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:15.187508 23590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:48:15.187543 23590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:15.197731 23590 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.131:45667
I20260806 02:48:15.197799 23783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.131:45667 every 8 connection(s)
I20260806 02:48:15.198846 23590 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
I20260806 02:48:15.207933 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 23590
Aug 06 02:48:15 dist-test-slave-gmjp krb5kdc[22240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984495, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:48:15.214520 23393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:15.200883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:46305 (local address 127.1.183.190:46301)
0806 02:48:15.201038 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:15.201042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:15.201982 (+ 940us) server_negotiation.cc:407] Connection header received
0806 02:48:15.203392 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:15.203395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:15.203461 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:15.203568 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:15.204904 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.205434 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.206664 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.206890 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.207002 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:15.209984 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:15.209987 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:15.209990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:15.210016 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:15.211928 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:15.212642 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:15.212646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:15.212647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:15.212743 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:15.213146 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:15.213149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:15.213150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:15.213448 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:15.213547 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:15.214231 (+ 684us) server_negotiation.cc:299] Negotiation successful
0806 02:48:15.214388 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":62}
I20260806 02:48:15.215730 23789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:15.201279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.131:46305)
0806 02:48:15.201805 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:15.201854 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:15.203129 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:15.203764 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:15.203771 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:15.204218 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:15.204744 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:15.204758 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.205587 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.205591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:15.206516 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:15.206525 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.207062 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.207066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:15.207240 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:15.207950 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:15.207967 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:15.209773 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:15.212059 (+ 2286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:15.212064 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:15.212078 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:15.212521 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:15.212849 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:15.212852 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:15.212854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:15.213045 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:15.213557 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:15.213563 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:15.213898 (+ 335us) client_negotiation.cc:769] Sending connection context
0806 02:48:15.214143 (+ 245us) client_negotiation.cc:240] Negotiation successful
0806 02:48:15.214551 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":126,"threads_started":1}
I20260806 02:48:15.217109 23784 heartbeater.cc:347] Connected to a master server at 127.1.183.190:46301
I20260806 02:48:15.217399 23784 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:15.218032 23784 heartbeater.cc:511] Master 127.1.183.190:46301 requested a full tablet report, sending...
I20260806 02:48:15.219457 23106 ts_manager.cc:194] Registered new tserver with Master: 48b3dfeef0cd4ba4b711d09bc2410f4a (127.1.183.131:45667)
I20260806 02:48:15.219457 23791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:15.208794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40552 (local address 127.1.183.190:46301)
0806 02:48:15.209093 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:15.209097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:15.209209 (+ 112us) server_negotiation.cc:407] Connection header received
0806 02:48:15.209991 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:15.209995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:15.210050 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:15.210158 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:15.211116 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.212053 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.213274 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.213646 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.213752 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:15.215363 (+ 1611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:15.215366 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:15.215368 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:15.215390 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:15.217209 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:15.217751 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:15.217756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:15.217758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:15.217849 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:15.218305 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:15.218309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:15.218311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:15.218566 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:15.218829 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:15.219089 (+ 260us) server_negotiation.cc:299] Negotiation successful
0806 02:48:15.219237 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":231,"thread_start_us":121,"threads_started":1}
I20260806 02:48:15.220082 23106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.131@KRBTEST.COM'} at 127.1.183.131:46305
I20260806 02:48:15.221433 1758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:48:15.225903 23791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:15.222071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40560 (local address 127.1.183.190:46301)
0806 02:48:15.222193 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:15.222198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:15.222489 (+ 291us) server_negotiation.cc:407] Connection header received
0806 02:48:15.222614 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:15.222617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:15.222703 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:15.222786 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:48:15.223056 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.223542 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.224537 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.224759 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.224837 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:15.225139 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:48:15.225503 (+ 364us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:48:15.225591 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:15.225731 (+ 140us) server_negotiation.cc:299] Negotiation successful
0806 02:48:15.225785 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 02:48:15.226576 23106 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:40560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:48:15.230151 23106 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:48:15.238737 23298 tablet_service.cc:1467] Processing CreateTablet for tablet d7d14531c9de4d2484ef72dd2abea67f (DEFAULT_TABLE table=test-table [id=6890db045b8b4c21b74c0ec2e58ae686]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:48:15.239472 23298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7d14531c9de4d2484ef72dd2abea67f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:48:15.242674 23220 tablet_bootstrap.cc:492] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977: Bootstrap starting.
I20260806 02:48:15.242977 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:15.237264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:45667 (local address 127.0.0.1:41596)
0806 02:48:15.237380 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:15.237399 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:15.237492 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:15.238563 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:15.238565 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:15.238581 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:15.238671 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:15.238675 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.240040 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.240043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:15.241310 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:15.241321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.242532 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.242536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:15.242634 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:15.242682 (+ 48us) client_negotiation.cc:769] Sending connection context
0806 02:48:15.242754 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 02:48:15.242828 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260806 02:48:15.243599 23797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:15.237506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41596 (local address 127.1.183.131:45667)
0806 02:48:15.238007 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:15.238025 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:15.238055 (+ 30us) server_negotiation.cc:407] Connection header received
0806 02:48:15.238140 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:15.238144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:15.238387 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:15.238576 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:15.238760 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.239894 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.241458 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.242363 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.242496 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:15.242678 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:15.243357 (+ 679us) server_negotiation.cc:299] Negotiation successful
0806 02:48:15.243461 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":161,"threads_started":1}
I20260806 02:48:15.243690 23220 tablet_bootstrap.cc:654] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:15.245270 23220 tablet_bootstrap.cc:492] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977: No bootstrap required, opened a new log
I20260806 02:48:15.245302 23799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:15.237572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:37461 (local address 127.0.0.1:58140)
0806 02:48:15.237833 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:15.237847 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:15.237932 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:15.241390 (+ 3458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:15.241396 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:15.241417 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:15.241508 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:15.241512 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.242558 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.242560 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:15.243592 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:15.243605 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.244908 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.244912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:15.245008 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:15.245033 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 02:48:15.245104 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 02:48:15.245174 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":135,"threads_started":1}
I20260806 02:48:15.245395 23220 ts_tablet_manager.cc:1397] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260806 02:48:15.245783 23800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:15.238056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58140 (local address 127.1.183.130:37461)
0806 02:48:15.240596 (+ 2540us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:15.240607 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:15.240631 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:48:15.240705 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:15.240710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:15.241063 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:15.241213 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:15.241631 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.242374 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.243738 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.244588 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.244697 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:15.244841 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:15.245538 (+ 697us) server_negotiation.cc:299] Negotiation successful
0806 02:48:15.245644 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2428,"thread_start_us":131,"threads_started":1}
I20260806 02:48:15.245891 23220 raft_consensus.cc:348] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45667 } } peers { permanent_uuid: "5d733086cd7249c08b869c31d708b865" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37461 } } peers { permanent_uuid: "ebe3e718f749433d879f26d19ec8e977" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40057 } }
I20260806 02:48:15.246027 23220 raft_consensus.cc:374] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:15.246069 23220 raft_consensus.cc:729] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebe3e718f749433d879f26d19ec8e977, State: Initialized, Role: FOLLOWER
I20260806 02:48:15.246265 23220 consensus_queue.cc:260] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45667 } } peers { permanent_uuid: "5d733086cd7249c08b869c31d708b865" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37461 } } peers { permanent_uuid: "ebe3e718f749433d879f26d19ec8e977" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40057 } }
I20260806 02:48:15.247105 23684 tablet_service.cc:1467] Processing CreateTablet for tablet d7d14531c9de4d2484ef72dd2abea67f (DEFAULT_TABLE table=test-table [id=6890db045b8b4c21b74c0ec2e58ae686]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:48:15.247717 23220 ts_tablet_manager.cc:1428] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 02:48:15.248576 23684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7d14531c9de4d2484ef72dd2abea67f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:48:15.248728 23494 tablet_service.cc:1467] Processing CreateTablet for tablet d7d14531c9de4d2484ef72dd2abea67f (DEFAULT_TABLE table=test-table [id=6890db045b8b4c21b74c0ec2e58ae686]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:48:15.249776 23494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7d14531c9de4d2484ef72dd2abea67f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:48:15.257524 23803 tablet_bootstrap.cc:492] T d7d14531c9de4d2484ef72dd2abea67f P 48b3dfeef0cd4ba4b711d09bc2410f4a: Bootstrap starting.
I20260806 02:48:15.257524 23804 tablet_bootstrap.cc:492] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865: Bootstrap starting.
I20260806 02:48:15.260584 23803 tablet_bootstrap.cc:654] T d7d14531c9de4d2484ef72dd2abea67f P 48b3dfeef0cd4ba4b711d09bc2410f4a: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:15.260584 23804 tablet_bootstrap.cc:654] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:15.261561 23804 log.cc:824] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:15.261562 23803 log.cc:824] T d7d14531c9de4d2484ef72dd2abea67f P 48b3dfeef0cd4ba4b711d09bc2410f4a: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:15.263775 23803 tablet_bootstrap.cc:492] T d7d14531c9de4d2484ef72dd2abea67f P 48b3dfeef0cd4ba4b711d09bc2410f4a: No bootstrap required, opened a new log
I20260806 02:48:15.263767 23804 tablet_bootstrap.cc:492] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865: No bootstrap required, opened a new log
I20260806 02:48:15.263979 23803 ts_tablet_manager.cc:1397] T d7d14531c9de4d2484ef72dd2abea67f P 48b3dfeef0cd4ba4b711d09bc2410f4a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 02:48:15.263979 23804 ts_tablet_manager.cc:1397] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 02:48:15.268112 23804 raft_consensus.cc:348] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45667 } } peers { permanent_uuid: "5d733086cd7249c08b869c31d708b865" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37461 } } peers { permanent_uuid: "ebe3e718f749433d879f26d19ec8e977" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40057 } }
I20260806 02:48:15.268112 23803 raft_consensus.cc:348] T d7d14531c9de4d2484ef72dd2abea67f P 48b3dfeef0cd4ba4b711d09bc2410f4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45667 } } peers { permanent_uuid: "5d733086cd7249c08b869c31d708b865" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37461 } } peers { permanent_uuid: "ebe3e718f749433d879f26d19ec8e977" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40057 } }
I20260806 02:48:15.268455 23803 raft_consensus.cc:374] T d7d14531c9de4d2484ef72dd2abea67f P 48b3dfeef0cd4ba4b711d09bc2410f4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:15.268455 23804 raft_consensus.cc:374] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:15.268518 23803 raft_consensus.cc:729] T d7d14531c9de4d2484ef72dd2abea67f P 48b3dfeef0cd4ba4b711d09bc2410f4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48b3dfeef0cd4ba4b711d09bc2410f4a, State: Initialized, Role: FOLLOWER
I20260806 02:48:15.268527 23804 raft_consensus.cc:729] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d733086cd7249c08b869c31d708b865, State: Initialized, Role: FOLLOWER
I20260806 02:48:15.269027 23804 consensus_queue.cc:260] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45667 } } peers { permanent_uuid: "5d733086cd7249c08b869c31d708b865" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37461 } } peers { permanent_uuid: "ebe3e718f749433d879f26d19ec8e977" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40057 } }
I20260806 02:48:15.269026 23803 consensus_queue.cc:260] T d7d14531c9de4d2484ef72dd2abea67f P 48b3dfeef0cd4ba4b711d09bc2410f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45667 } } peers { permanent_uuid: "5d733086cd7249c08b869c31d708b865" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37461 } } peers { permanent_uuid: "ebe3e718f749433d879f26d19ec8e977" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40057 } }
I20260806 02:48:15.269766 23586 heartbeater.cc:503] Master 127.1.183.190:46301 was elected leader, sending a full tablet report...
I20260806 02:48:15.270031 23784 heartbeater.cc:503] Master 127.1.183.190:46301 was elected leader, sending a full tablet report...
I20260806 02:48:15.270153 23804 ts_tablet_manager.cc:1428] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260806 02:48:15.270153 23803 ts_tablet_manager.cc:1428] T d7d14531c9de4d2484ef72dd2abea67f P 48b3dfeef0cd4ba4b711d09bc2410f4a: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
W20260806 02:48:15.288764 23391 tablet.cc:2369] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:48:15.449965 23785 tablet.cc:2369] T d7d14531c9de4d2484ef72dd2abea67f P 48b3dfeef0cd4ba4b711d09bc2410f4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:48:15.488348 23588 tablet.cc:2369] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:48:15.666339 23809 raft_consensus.cc:482] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:48:15.666607 23809 raft_consensus.cc:504] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45667 } } peers { permanent_uuid: "5d733086cd7249c08b869c31d708b865" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37461 } } peers { permanent_uuid: "ebe3e718f749433d879f26d19ec8e977" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40057 } }
I20260806 02:48:15.667932 23809 leader_election.cc:288] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48b3dfeef0cd4ba4b711d09bc2410f4a (127.1.183.131:45667), ebe3e718f749433d879f26d19ec8e977 (127.1.183.129:40057)
I20260806 02:48:15.668244 23397 raft_consensus.cc:482] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:48:15.668473 23397 raft_consensus.cc:504] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45667 } } peers { permanent_uuid: "5d733086cd7249c08b869c31d708b865" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37461 } } peers { permanent_uuid: "ebe3e718f749433d879f26d19ec8e977" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40057 } }
I20260806 02:48:15.669739 23397 leader_election.cc:288] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48b3dfeef0cd4ba4b711d09bc2410f4a (127.1.183.131:45667), 5d733086cd7249c08b869c31d708b865 (127.1.183.130:37461)
I20260806 02:48:15.673152 23797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:15.668406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:49213 (local address 127.1.183.131:45667)
0806 02:48:15.668641 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:15.668646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:15.668860 (+ 214us) server_negotiation.cc:407] Connection header received
0806 02:48:15.668956 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:15.668960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:15.669015 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:15.669107 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:15.669488 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.670232 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.671802 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.672416 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.672521 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:15.672580 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:15.672931 (+ 351us) server_negotiation.cc:299] Negotiation successful
0806 02:48:15.673018 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260806 02:48:15.673650 23811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:15.668219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:45667 (local address 127.1.183.130:49213)
0806 02:48:15.668760 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:15.668774 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:15.668864 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:15.669167 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:15.669171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:15.669198 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:15.669338 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:15.669346 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.670372 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.670376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:15.671644 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:15.671657 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.672687 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.672691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:15.672795 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:15.672817 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:48:15.673361 (+ 544us) client_negotiation.cc:240] Negotiation successful
0806 02:48:15.673477 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":61,"thread_start_us":60,"threads_started":1}
I20260806 02:48:15.673940 23813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:15.668950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:47565 (local address 127.1.183.129:40057)
0806 02:48:15.669302 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:15.669307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:15.669321 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:48:15.669370 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:15.669373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:15.669438 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:15.669540 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:15.669901 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.670637 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.672137 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.672871 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.672981 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:15.673119 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:15.673708 (+ 589us) server_negotiation.cc:299] Negotiation successful
0806 02:48:15.673802 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":280,"thread_start_us":167,"threads_started":1}
I20260806 02:48:15.673662 23812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:15.668215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:40057 (local address 127.1.183.130:47565)
0806 02:48:15.668600 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:15.668617 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:15.668744 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:15.669549 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:15.669552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:15.669568 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:15.669702 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:15.669709 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.670766 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.670772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:15.672006 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:15.672016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.673183 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.673186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:15.673315 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:15.673339 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 02:48:15.673407 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 02:48:15.673493 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":2,"thread_start_us":107,"threads_started":1}
I20260806 02:48:15.674602 23719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7d14531c9de4d2484ef72dd2abea67f" candidate_uuid: "5d733086cd7249c08b869c31d708b865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" is_pre_election: true
I20260806 02:48:15.674715 23815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:15.669906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:45667 (local address 127.1.183.129:59841)
0806 02:48:15.670403 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:15.670416 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:15.670500 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:15.670990 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:15.670993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:15.671013 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:15.671097 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:15.671102 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.672184 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.672188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:15.673371 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:15.673382 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.674278 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.674283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:15.674398 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:15.674421 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 02:48:15.674493 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 02:48:15.674566 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":117,"threads_started":1}
I20260806 02:48:15.674965 23817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:15.670294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:59841 (local address 127.1.183.131:45667)
0806 02:48:15.670733 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:15.670747 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:15.670760 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:48:15.670802 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:15.670805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:15.670846 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:15.671189 (+ 343us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:15.671282 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.672045 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.673522 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.674114 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.674286 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:15.674338 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:15.674550 (+ 212us) server_negotiation.cc:299] Negotiation successful
0806 02:48:15.674628 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":81,"threads_started":1}
I20260806 02:48:15.674957 23719 raft_consensus.cc:2454] T d7d14531c9de4d2484ef72dd2abea67f P 48b3dfeef0cd4ba4b711d09bc2410f4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d733086cd7249c08b869c31d708b865 in term 0.
I20260806 02:48:15.675367 23323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7d14531c9de4d2484ef72dd2abea67f" candidate_uuid: "5d733086cd7249c08b869c31d708b865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebe3e718f749433d879f26d19ec8e977" is_pre_election: true
I20260806 02:48:15.675573 23719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7d14531c9de4d2484ef72dd2abea67f" candidate_uuid: "ebe3e718f749433d879f26d19ec8e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" is_pre_election: true
I20260806 02:48:15.675606 23323 raft_consensus.cc:2454] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d733086cd7249c08b869c31d708b865 in term 0.
I20260806 02:48:15.675691 23719 raft_consensus.cc:2454] T d7d14531c9de4d2484ef72dd2abea67f P 48b3dfeef0cd4ba4b711d09bc2410f4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebe3e718f749433d879f26d19ec8e977 in term 0.
I20260806 02:48:15.675689 23426 leader_election.cc:302] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48b3dfeef0cd4ba4b711d09bc2410f4a, 5d733086cd7249c08b869c31d708b865; no voters:
I20260806 02:48:15.675967 23814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:15.670042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:37461 (local address 127.1.183.129:51693)
0806 02:48:15.670894 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:15.670907 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:15.671029 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:15.671351 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:15.671353 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:15.671382 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:15.671460 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:15.671465 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.672390 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.672393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:15.673697 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:15.674387 (+ 690us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.675323 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.675326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:15.675429 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:15.675464 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 02:48:15.675714 (+ 250us) client_negotiation.cc:240] Negotiation successful
0806 02:48:15.675788 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"mutex_wait_us":4,"thread_start_us":66,"threads_started":1}
I20260806 02:48:15.676023 23809 raft_consensus.cc:2781] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:48:15.676137 23809 raft_consensus.cc:482] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:48:15.676221 23809 raft_consensus.cc:3037] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:15.676357 23800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:15.670516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:51693 (local address 127.1.183.130:37461)
0806 02:48:15.670617 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:15.670622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:15.670968 (+ 346us) server_negotiation.cc:407] Connection header received
0806 02:48:15.671135 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:15.671138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:15.671185 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:15.671233 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:15.671573 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.672252 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.674508 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.675055 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.675145 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:15.675206 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:15.675585 (+ 379us) server_negotiation.cc:299] Negotiation successful
0806 02:48:15.675665 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 02:48:15.676494 23211 leader_election.cc:302] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48b3dfeef0cd4ba4b711d09bc2410f4a, ebe3e718f749433d879f26d19ec8e977; no voters:
I20260806 02:48:15.676764 23397 raft_consensus.cc:2781] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:48:15.676860 23397 raft_consensus.cc:482] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:48:15.676919 23397 raft_consensus.cc:3037] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:15.676889 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7d14531c9de4d2484ef72dd2abea67f" candidate_uuid: "ebe3e718f749433d879f26d19ec8e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d733086cd7249c08b869c31d708b865" is_pre_election: true
I20260806 02:48:15.677608 23809 raft_consensus.cc:504] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45667 } } peers { permanent_uuid: "5d733086cd7249c08b869c31d708b865" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37461 } } peers { permanent_uuid: "ebe3e718f749433d879f26d19ec8e977" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40057 } }
I20260806 02:48:15.677910 23518 raft_consensus.cc:2379] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ebe3e718f749433d879f26d19ec8e977 in current term 1: Already voted for candidate 5d733086cd7249c08b869c31d708b865 in this term.
I20260806 02:48:15.678122 23809 leader_election.cc:288] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865 [CANDIDATE]: Term 1 election: Requested vote from peers 48b3dfeef0cd4ba4b711d09bc2410f4a (127.1.183.131:45667), ebe3e718f749433d879f26d19ec8e977 (127.1.183.129:40057)
I20260806 02:48:15.678187 23397 raft_consensus.cc:504] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45667 } } peers { permanent_uuid: "5d733086cd7249c08b869c31d708b865" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37461 } } peers { permanent_uuid: "ebe3e718f749433d879f26d19ec8e977" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40057 } }
I20260806 02:48:15.678555 23323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7d14531c9de4d2484ef72dd2abea67f" candidate_uuid: "5d733086cd7249c08b869c31d708b865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebe3e718f749433d879f26d19ec8e977"
I20260806 02:48:15.678752 23323 raft_consensus.cc:2379] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5d733086cd7249c08b869c31d708b865 in current term 1: Already voted for candidate ebe3e718f749433d879f26d19ec8e977 in this term.
I20260806 02:48:15.678978 23719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7d14531c9de4d2484ef72dd2abea67f" candidate_uuid: "5d733086cd7249c08b869c31d708b865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a"
I20260806 02:48:15.679059 23718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7d14531c9de4d2484ef72dd2abea67f" candidate_uuid: "ebe3e718f749433d879f26d19ec8e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a"
I20260806 02:48:15.679119 23719 raft_consensus.cc:3037] T d7d14531c9de4d2484ef72dd2abea67f P 48b3dfeef0cd4ba4b711d09bc2410f4a [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:15.678603 23397 leader_election.cc:288] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [CANDIDATE]: Term 1 election: Requested vote from peers 48b3dfeef0cd4ba4b711d09bc2410f4a (127.1.183.131:45667), 5d733086cd7249c08b869c31d708b865 (127.1.183.130:37461)
I20260806 02:48:15.679678 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7d14531c9de4d2484ef72dd2abea67f" candidate_uuid: "ebe3e718f749433d879f26d19ec8e977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d733086cd7249c08b869c31d708b865"
I20260806 02:48:15.679855 23518 raft_consensus.cc:2379] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ebe3e718f749433d879f26d19ec8e977 in current term 1: Already voted for candidate 5d733086cd7249c08b869c31d708b865 in this term.
I20260806 02:48:15.680184 23217 leader_election.cc:302] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebe3e718f749433d879f26d19ec8e977; no voters: 48b3dfeef0cd4ba4b711d09bc2410f4a, 5d733086cd7249c08b869c31d708b865
I20260806 02:48:15.680365 23397 raft_consensus.cc:2726] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 02:48:15.680524 23719 raft_consensus.cc:2454] T d7d14531c9de4d2484ef72dd2abea67f P 48b3dfeef0cd4ba4b711d09bc2410f4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d733086cd7249c08b869c31d708b865 in term 1.
I20260806 02:48:15.680823 23426 leader_election.cc:302] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48b3dfeef0cd4ba4b711d09bc2410f4a, 5d733086cd7249c08b869c31d708b865; no voters: ebe3e718f749433d879f26d19ec8e977
I20260806 02:48:15.680998 23809 raft_consensus.cc:2781] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:48:15.681224 23809 raft_consensus.cc:686] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865 [term 1 LEADER]: Becoming Leader. State: Replica: 5d733086cd7249c08b869c31d708b865, State: Running, Role: LEADER
I20260806 02:48:15.681577 23809 consensus_queue.cc:237] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45667 } } peers { permanent_uuid: "5d733086cd7249c08b869c31d708b865" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37461 } } peers { permanent_uuid: "ebe3e718f749433d879f26d19ec8e977" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40057 } }
I20260806 02:48:15.685075 23106 catalog_manager.cc:5495] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d733086cd7249c08b869c31d708b865 (127.1.183.130). New cstate: current_term: 1 leader_uuid: "5d733086cd7249c08b869c31d708b865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d733086cd7249c08b869c31d708b865" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebe3e718f749433d879f26d19ec8e977" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40057 } health_report { overall_health: UNKNOWN } } }
I20260806 02:48:15.699361 23800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:15.695910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58144 (local address 127.1.183.130:37461)
0806 02:48:15.696048 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:15.696052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:15.696083 (+ 31us) server_negotiation.cc:407] Connection header received
0806 02:48:15.696195 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:15.696197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:15.696256 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:15.696341 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:48:15.696572 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.697078 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.697897 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:15.698113 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:15.698191 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:15.698554 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:48:15.698999 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:48:15.699092 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:15.699191 (+ 99us) server_negotiation.cc:299] Negotiation successful
0806 02:48:15.699241 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:48:15.705806 23323 raft_consensus.cc:1261] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [term 1 FOLLOWER]: Refusing update from remote peer 5d733086cd7249c08b869c31d708b865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:48:15.705807 23719 raft_consensus.cc:1261] T d7d14531c9de4d2484ef72dd2abea67f P 48b3dfeef0cd4ba4b711d09bc2410f4a [term 1 FOLLOWER]: Refusing update from remote peer 5d733086cd7249c08b869c31d708b865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:48:15.706522 23818 consensus_queue.cc:1035] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebe3e718f749433d879f26d19ec8e977" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:48:15.706720 23809 consensus_queue.cc:1035] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:48:15.714155 23826 mvcc.cc:205] Tried to move back new op lower bound from 7315392494402990080 to 7315392494314295296. Current Snapshot: MvccSnapshot[applied={T|T < 7315392494402990080}]
I20260806 02:48:15.724723 23104 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:40560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:48:15.724921 23104 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:40560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:48:15.727926 23104 catalog_manager.cc:5782] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2: Sending DeleteTablet for 3 replicas of tablet d7d14531c9de4d2484ef72dd2abea67f
I20260806 02:48:15.728863 23298 tablet_service.cc:1514] Processing DeleteTablet for tablet d7d14531c9de4d2484ef72dd2abea67f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:48:15 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:41194
I20260806 02:48:15.728941 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 23162
I20260806 02:48:15.728942 23684 tablet_service.cc:1514] Processing DeleteTablet for tablet d7d14531c9de4d2484ef72dd2abea67f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:48:15 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:41596
I20260806 02:48:15.733584 23839 tablet_replica.cc:330] T d7d14531c9de4d2484ef72dd2abea67f P 48b3dfeef0cd4ba4b711d09bc2410f4a: stopping tablet replica
I20260806 02:48:15.733712 23494 tablet_service.cc:1514] Processing DeleteTablet for tablet d7d14531c9de4d2484ef72dd2abea67f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:48:15 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:58140
I20260806 02:48:15.734015 23839 raft_consensus.cc:2229] T d7d14531c9de4d2484ef72dd2abea67f P 48b3dfeef0cd4ba4b711d09bc2410f4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:48:15.734362 23840 tablet_replica.cc:330] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865: stopping tablet replica
I20260806 02:48:15.734408 23839 raft_consensus.cc:2258] T d7d14531c9de4d2484ef72dd2abea67f P 48b3dfeef0cd4ba4b711d09bc2410f4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:48:15.734756 23840 raft_consensus.cc:2229] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:48:15.735231 23840 raft_consensus.cc:2258] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:48:15.738994 23840 ts_tablet_manager.cc:1905] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:48:15.739496 23839 ts_tablet_manager.cc:1905] T d7d14531c9de4d2484ef72dd2abea67f P 48b3dfeef0cd4ba4b711d09bc2410f4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:48:15.742103 23840 ts_tablet_manager.cc:1918] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:48:15.742288 23840 log.cc:1196] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/wal/wals/d7d14531c9de4d2484ef72dd2abea67f
I20260806 02:48:15.742719 23840 ts_tablet_manager.cc:1939] T d7d14531c9de4d2484ef72dd2abea67f P 5d733086cd7249c08b869c31d708b865: Deleting consensus metadata
I20260806 02:48:15.742998 23839 ts_tablet_manager.cc:1918] T d7d14531c9de4d2484ef72dd2abea67f P 48b3dfeef0cd4ba4b711d09bc2410f4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:48:15.743111 23839 log.cc:1196] T d7d14531c9de4d2484ef72dd2abea67f P 48b3dfeef0cd4ba4b711d09bc2410f4a: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/wal/wals/d7d14531c9de4d2484ef72dd2abea67f
I20260806 02:48:15.743443 23839 ts_tablet_manager.cc:1939] T d7d14531c9de4d2484ef72dd2abea67f P 48b3dfeef0cd4ba4b711d09bc2410f4a: Deleting consensus metadata
I20260806 02:48:15.744445 23083 catalog_manager.cc:4841] TS 5d733086cd7249c08b869c31d708b865 (127.1.183.130:37461): tablet d7d14531c9de4d2484ef72dd2abea67f (table test-table [id=6890db045b8b4c21b74c0ec2e58ae686]) successfully deleted
I20260806 02:48:15.744771 23076 catalog_manager.cc:4841] TS 48b3dfeef0cd4ba4b711d09bc2410f4a (127.1.183.131:45667): tablet d7d14531c9de4d2484ef72dd2abea67f (table test-table [id=6890db045b8b4c21b74c0ec2e58ae686]) successfully deleted
I20260806 02:48:15.745851 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 23392
W20260806 02:48:15.745886 23078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.183.129:40057 (error 108)
I20260806 02:48:15.749302 23842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:15.748730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:40057 (local address 127.0.0.1:41196)
0806 02:48:15.749058 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:15.749157 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.129:40057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":63,"threads_started":1}
W20260806 02:48:15.749480 23078 catalog_manager.cc:4584] TS ebe3e718f749433d879f26d19ec8e977 (127.1.183.129:40057): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d7d14531c9de4d2484ef72dd2abea67f: Network error: Client connection negotiation failed: client connection to 127.1.183.129:40057: connect: Connection refused (error 111)
I20260806 02:48:15.757701 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 23590
I20260806 02:48:15.773721 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 23053
I20260806 02:48:15.786208 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:46301
--webserver_interface=127.1.183.190
--webserver_port=43579
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:38541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:46301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:48:15.904022 23843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:15.904469 23843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:15.904567 23843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:15.908772 23843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:48:15.908893 23843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:15.908927 23843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:15.908959 23843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:48:15.908998 23843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:48:15.914651 23843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:38541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:46301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:46301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=43579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:48:15.915874 23843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:15.917004 23843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:15.925933 23851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:15.925994 23850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:15.925981 23853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:15.926210 23843 server_base.cc:1034] running on GCE node
I20260806 02:48:15.926849 23843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:15.928042 23843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:15.929240 23843 hybrid_clock.cc:648] HybridClock initialized: now 1785984495929224 us; error 28 us; skew 500 ppm
Aug 06 02:48:15 dist-test-slave-gmjp krb5kdc[22240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984495, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:15.932209 23843 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:48:15.933209 23843 webserver.cc:460] Webserver started at http://127.1.183.190:43579/ using document root <none> and password file <none>
I20260806 02:48:15.933804 23843 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:15.933902 23843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:15.937970 23843 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:48:15.940318 23862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:15.941589 23843 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:48:15.941741 23843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "efc9d63000364ad9a38827aededd20e2"
format_stamp: "Formatted at 2026-08-06 02:48:12 on dist-test-slave-gmjp"
server_key: "4526a6d2f8b50564bde8aae1fcdf5fb7"
server_key_iv: "9b3c512375fd98816072daaee70c5ec7"
server_key_version: "encryptionkey@0"
I20260806 02:48:15.942380 23843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:15.959834 23843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:15.963539 23843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:15.963773 23843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:15.971366 23843 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:46301
I20260806 02:48:15.971669 23941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:46301 every 8 connection(s)
I20260806 02:48:15.972653 23843 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:48:15.974179 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 23843
I20260806 02:48:15.974689 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.129:40057
--local_ip_for_outbound_sockets=127.1.183.129
--tserver_master_addrs=127.1.183.190:46301
--webserver_port=40849
--webserver_interface=127.1.183.129
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:38541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:48:15.978048 23942 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:48:15.980012 23942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2: Bootstrap starting.
I20260806 02:48:15.998036 23942 log.cc:824] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:16.009760 23942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:48:16.010227 23942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2: Bootstrap complete.
I20260806 02:48:16.014497 23942 raft_consensus.cc:348] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc9d63000364ad9a38827aededd20e2" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 46301 } }
I20260806 02:48:16.014916 23942 raft_consensus.cc:729] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: efc9d63000364ad9a38827aededd20e2, State: Initialized, Role: FOLLOWER
I20260806 02:48:16.015436 23942 consensus_queue.cc:260] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc9d63000364ad9a38827aededd20e2" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 46301 } }
I20260806 02:48:16.015556 23942 raft_consensus.cc:388] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:48:16.015641 23942 raft_consensus.cc:482] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:48:16.015734 23942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 02:48:16.017812 23942 raft_consensus.cc:504] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc9d63000364ad9a38827aededd20e2" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 46301 } }
I20260806 02:48:16.018158 23942 leader_election.cc:302] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efc9d63000364ad9a38827aededd20e2; no voters:
I20260806 02:48:16.018540 23942 leader_election.cc:288] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 02:48:16.018635 23949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 02:48:16.018846 23949 raft_consensus.cc:686] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [term 3 LEADER]: Becoming Leader. State: Replica: efc9d63000364ad9a38827aededd20e2, State: Running, Role: LEADER
I20260806 02:48:16.019162 23949 consensus_queue.cc:237] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc9d63000364ad9a38827aededd20e2" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 46301 } }
I20260806 02:48:16.019860 23942 sys_catalog.cc:564] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:48:16.020685 23950 sys_catalog.cc:455] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "efc9d63000364ad9a38827aededd20e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc9d63000364ad9a38827aededd20e2" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 46301 } } }
I20260806 02:48:16.020723 23951 sys_catalog.cc:455] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efc9d63000364ad9a38827aededd20e2. Latest consensus state: current_term: 3 leader_uuid: "efc9d63000364ad9a38827aededd20e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc9d63000364ad9a38827aededd20e2" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 46301 } } }
I20260806 02:48:16.020816 23950 sys_catalog.cc:458] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [sys.catalog]: This master's current role is: LEADER
I20260806 02:48:16.020825 23951 sys_catalog.cc:458] T 00000000000000000000000000000000 P efc9d63000364ad9a38827aededd20e2 [sys.catalog]: This master's current role is: LEADER
I20260806 02:48:16.021178 23953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:48:16.026676 23953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:48:16.027395 23953 catalog_manager.cc:1259] Loaded cluster ID: 7ca99d4d402d42278037be43e9849523
I20260806 02:48:16.027464 23953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260806 02:48:16.028594 23953 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0facb699ad google::LogMessage::Fail() at ??:0
@ 0x7f0facb6c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f0facb694de google::LogMessage::Flush() at ??:0
@ 0x7f0facb6aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0fb74a8fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f0fb74a9d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0fb74a6579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f0fb74ea640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f0faedbb3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0faedbbc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f0faedbd3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f0faedac512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0fb03f06db start_thread at ??:0
@ 0x7f0fab66761f clone at ??:0
W20260806 02:48:16.127105 23946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:16.127401 23946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:16.127458 23946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:16.131685 23946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:16.131769 23946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:16.131831 23946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.129
I20260806 02:48:16.137559 23946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:38541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.129:40057
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.183.129
--webserver_port=40849
--tserver_master_addrs=127.1.183.190:46301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.129
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:48:16.138816 23946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:16.139962 23946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:16.150347 24011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:16.150347 24006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:16.150346 24009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:16.150943 23946 server_base.cc:1034] running on GCE node
I20260806 02:48:16.151413 23946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:16.152220 23946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:16.153411 23946 hybrid_clock.cc:648] HybridClock initialized: now 1785984496153379 us; error 59 us; skew 500 ppm
Aug 06 02:48:16 dist-test-slave-gmjp krb5kdc[22240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984496, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:16.156446 23946 init.cc:377] Logged in from keytab as kudu/127.1.183.129@KRBTEST.COM (short username kudu)
I20260806 02:48:16.157541 23946 webserver.cc:460] Webserver started at http://127.1.183.129:40849/ using document root <none> and password file <none>
I20260806 02:48:16.158150 23946 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:16.158215 23946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:16.162515 23946 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:48:16.165005 24020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:16.166543 23946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:48:16.166682 23946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/wal
uuid: "ebe3e718f749433d879f26d19ec8e977"
format_stamp: "Formatted at 2026-08-06 02:48:13 on dist-test-slave-gmjp"
server_key: "dc2ec2dde1a60ae694d399cc8137925d"
server_key_iv: "35ab59c11508b49a521ad83ac33ad786"
server_key_version: "encryptionkey@0"
I20260806 02:48:16.167276 23946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:16.184021 23946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:16.187875 23946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:16.188097 23946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:16.190276 23946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:48:16.191627 24028 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 02:48:16.196055 23946 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 02:48:16.196127 23946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260806 02:48:16.196236 23946 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 02:48:16.198009 23946 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 02:48:16.198072 23946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 02:48:16.198349 24028 tablet_bootstrap.cc:492] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977: Bootstrap starting.
I20260806 02:48:16.209527 23946 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.129:40057
I20260806 02:48:16.209830 24184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.129:40057 every 8 connection(s)
I20260806 02:48:16.210987 23946 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
I20260806 02:48:16.213874 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 23946
I20260806 02:48:16.214397 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.130:37461
--local_ip_for_outbound_sockets=127.1.183.130
--tserver_master_addrs=127.1.183.190:46301
--webserver_port=39981
--webserver_interface=127.1.183.130
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:38541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:48:16.221591 24028 log.cc:824] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:16.232776 24028 tablet_bootstrap.cc:492] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:48:16.233264 24028 tablet_bootstrap.cc:492] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977: Bootstrap complete.
I20260806 02:48:16.234947 24028 ts_tablet_manager.cc:1397] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977: Time spent bootstrapping tablet: real 0.037s user 0.029s sys 0.004s
I20260806 02:48:16.238433 24028 raft_consensus.cc:348] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45667 } } peers { permanent_uuid: "5d733086cd7249c08b869c31d708b865" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37461 } } peers { permanent_uuid: "ebe3e718f749433d879f26d19ec8e977" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40057 } }
I20260806 02:48:16.239187 24028 raft_consensus.cc:729] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebe3e718f749433d879f26d19ec8e977, State: Initialized, Role: FOLLOWER
I20260806 02:48:16.239961 24028 consensus_queue.cc:260] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45667 } } peers { permanent_uuid: "5d733086cd7249c08b869c31d708b865" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37461 } } peers { permanent_uuid: "ebe3e718f749433d879f26d19ec8e977" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40057 } }
I20260806 02:48:16.243048 24028 ts_tablet_manager.cc:1428] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
W20260806 02:48:16.248979 24188 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:48:16.213632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.129:48323)
0806 02:48:16.215069 (+ 1437us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:16.215133 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:16.216466 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:16.247971 (+ 31505us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1185,"thread_start_us":196,"threads_started":1}
W20260806 02:48:16.249541 24185 heartbeater.cc:644] Failed to heartbeat to 127.1.183.190:46301 (0 consecutive failures): Network error: Failed to ping master at 127.1.183.190:46301: Client connection negotiation failed: client connection to 127.1.183.190:46301: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 02:48:16.250339 24188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:16.250081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.129:48389)
0806 02:48:16.250217 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:16.250284 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 02:48:16.251063 24188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:16.250866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.129:44935)
0806 02:48:16.250974 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:16.251016 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 02:48:16.251312 24185 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:48:16.346927 24189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:16.347247 24189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:16.347324 24189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:16.351440 24189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:16.351547 24189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:16.351646 24189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.130
I20260806 02:48:16.357183 24189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:38541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.130:37461
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.183.130
--webserver_port=39981
--tserver_master_addrs=127.1.183.190:46301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.130
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:48:16.358471 24189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:16.359603 24189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:16.369758 24203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:16.369781 24201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:16.369762 24200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:16.370198 24189 server_base.cc:1034] running on GCE node
I20260806 02:48:16.370612 24189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:16.371321 24189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:16.372486 24189 hybrid_clock.cc:648] HybridClock initialized: now 1785984496372472 us; error 26 us; skew 500 ppm
Aug 06 02:48:16 dist-test-slave-gmjp krb5kdc[22240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984496, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:16.375445 24189 init.cc:377] Logged in from keytab as kudu/127.1.183.130@KRBTEST.COM (short username kudu)
I20260806 02:48:16.376493 24189 webserver.cc:460] Webserver started at http://127.1.183.130:39981/ using document root <none> and password file <none>
I20260806 02:48:16.377128 24189 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:16.377233 24189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:16.381845 24189 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260806 02:48:16.384275 24215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:16.385648 24189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:48:16.385802 24189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/wal
uuid: "5d733086cd7249c08b869c31d708b865"
format_stamp: "Formatted at 2026-08-06 02:48:13 on dist-test-slave-gmjp"
server_key: "38a71d57c4f6d1046ac832ef7dabeb58"
server_key_iv: "e72337064f96990dc798ce17925891cf"
server_key_version: "encryptionkey@0"
I20260806 02:48:16.386406 24189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:16.397415 24189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:16.401057 24189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:16.401285 24189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:16.401948 24189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:48:16.402850 24189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:48:16.402923 24189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:16.403013 24189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:48:16.403060 24189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:16.413240 24189 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.130:37461
I20260806 02:48:16.413277 24381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.130:37461 every 8 connection(s)
I20260806 02:48:16.414368 24189 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
I20260806 02:48:16.416415 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 24189
I20260806 02:48:16.416882 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.131:45667
--local_ip_for_outbound_sockets=127.1.183.131
--tserver_master_addrs=127.1.183.190:46301
--webserver_port=39195
--webserver_interface=127.1.183.131
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:38541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:48:16.419519 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:16.417022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.130:36689)
0806 02:48:16.417640 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:16.417819 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":121,"threads_started":1}
W20260806 02:48:16.420053 24382 heartbeater.cc:644] Failed to heartbeat to 127.1.183.190:46301 (0 consecutive failures): Network error: Failed to ping master at 127.1.183.190:46301: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
I20260806 02:48:16.420936 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:16.420642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.130:53935)
0806 02:48:16.420815 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:16.420869 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 02:48:16.421607 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:16.421392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.130:49477)
0806 02:48:16.421502 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:16.421556 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 02:48:16.421787 24382 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:48:16.536283 24386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:16.536650 24386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:16.536749 24386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:16.540982 24386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:16.541098 24386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:16.541203 24386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.131
I20260806 02:48:16.546801 24386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:38541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.131:45667
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.183.131
--webserver_port=39195
--tserver_master_addrs=127.1.183.190:46301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.131
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:48:16.548058 24386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:16.549188 24386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:16.559307 24396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:16.559309 24393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:16.559337 24392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:16.559723 24386 server_base.cc:1034] running on GCE node
I20260806 02:48:16.560117 24386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:16.560846 24386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:16.562045 24386 hybrid_clock.cc:648] HybridClock initialized: now 1785984496562015 us; error 55 us; skew 500 ppm
Aug 06 02:48:16 dist-test-slave-gmjp krb5kdc[22240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984496, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:16.569329 24386 init.cc:377] Logged in from keytab as kudu/127.1.183.131@KRBTEST.COM (short username kudu)
I20260806 02:48:16.570309 24386 webserver.cc:460] Webserver started at http://127.1.183.131:39195/ using document root <none> and password file <none>
I20260806 02:48:16.570904 24386 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:16.570967 24386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:16.575068 24386 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:48:16.577346 24405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:16.578711 24386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:48:16.578882 24386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/wal
uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a"
format_stamp: "Formatted at 2026-08-06 02:48:13 on dist-test-slave-gmjp"
server_key: "1e67b15636ba06b43f77b5e22e9b7307"
server_key_iv: "4f158e78b3ae16f15217bc0ab35947bb"
server_key_version: "encryptionkey@0"
I20260806 02:48:16.579445 24386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:16.597898 24386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:16.601223 24386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:16.601477 24386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:16.602165 24386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:48:16.603070 24386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:48:16.603118 24386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:16.603190 24386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:48:16.603230 24386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:16.613261 24386 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.131:45667
I20260806 02:48:16.613281 24565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.131:45667 every 8 connection(s)
I20260806 02:48:16.614406 24386 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
I20260806 02:48:16.616178 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 24386
I20260806 02:48:16.618956 24569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:16.617005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.131:47237)
0806 02:48:16.617571 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:16.617684 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":119,"threads_started":1}
W20260806 02:48:16.619508 24566 heartbeater.cc:644] Failed to heartbeat to 127.1.183.190:46301 (0 consecutive failures): Network error: Failed to ping master at 127.1.183.190:46301: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
I20260806 02:48:16.620332 24569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:16.619943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.131:50111)
0806 02:48:16.620108 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:16.620266 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260806 02:48:16.620978 24569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:16.620784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.131:60873)
0806 02:48:16.620885 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:16.620928 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 02:48:16.621222 24566 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 02:48:17.252486 24570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:17.251955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.129:56531)
0806 02:48:17.252297 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:17.252361 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":83,"threads_started":1}
I20260806 02:48:17.422959 24571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:17.422490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.130:53875)
0806 02:48:17.422774 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:17.422840 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":81,"threads_started":1}
I20260806 02:48:17.622396 24573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:17.621920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.131:38317)
0806 02:48:17.622209 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:17.622278 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":81,"threads_started":1}
I20260806 02:48:18.011085 24574 raft_consensus.cc:482] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:48:18.011384 24574 raft_consensus.cc:504] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45667 } } peers { permanent_uuid: "5d733086cd7249c08b869c31d708b865" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37461 } } peers { permanent_uuid: "ebe3e718f749433d879f26d19ec8e977" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40057 } }
I20260806 02:48:18.012738 24574 leader_election.cc:288] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 48b3dfeef0cd4ba4b711d09bc2410f4a (127.1.183.131:45667), 5d733086cd7249c08b869c31d708b865 (127.1.183.130:37461)
Aug 06 02:48:18 dist-test-slave-gmjp krb5kdc[22240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984496, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for kudu/127.1.183.130@KRBTEST.COM
Aug 06 02:48:18 dist-test-slave-gmjp krb5kdc[22240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984496, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for kudu/127.1.183.131@KRBTEST.COM
I20260806 02:48:18.027487 24577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:18.012907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:45667 (local address 127.1.183.129:44401)
0806 02:48:18.013238 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:18.013256 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:18.013382 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:18.015555 (+ 2173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:18.015561 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:18.015961 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:18.016509 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:18.016527 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:18.017837 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:18.017841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:18.018787 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:18.018797 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:18.019556 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:18.019560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:18.019805 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:18.020607 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:18.020619 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:18.022666 (+ 2047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:18.024978 (+ 2312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:18.024980 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:18.024982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:18.025245 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:18.025637 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:18.025640 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:18.025641 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:18.025799 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:18.026455 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:18.026462 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:18.026703 (+ 241us) client_negotiation.cc:769] Sending connection context
0806 02:48:18.026912 (+ 209us) client_negotiation.cc:240] Negotiation successful
0806 02:48:18.027280 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":156,"spinlock_wait_cycles":25728,"thread_start_us":92,"threads_started":1}
I20260806 02:48:18.027499 24580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:18.013232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:48901 (local address 127.1.183.130:37461)
0806 02:48:18.013630 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:18.013649 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:18.013696 (+ 47us) server_negotiation.cc:407] Connection header received
0806 02:48:18.014618 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:18.014628 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:18.015117 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:18.015481 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:18.016696 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:18.017686 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:18.018936 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:18.019402 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:18.019579 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:18.022583 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:18.022589 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:18.022602 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:18.022634 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:18.024651 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:18.025350 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:18.025357 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:18.025371 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:18.025503 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:18.025881 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:18.025887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:18.025889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:18.026335 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:18.026491 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:18.027026 (+ 535us) server_negotiation.cc:299] Negotiation successful
0806 02:48:18.027299 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":88,"threads_started":1}
I20260806 02:48:18.027534 24579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:18.013146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:44401 (local address 127.1.183.131:45667)
0806 02:48:18.013636 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:18.013657 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:18.013696 (+ 39us) server_negotiation.cc:407] Connection header received
0806 02:48:18.014618 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:18.014627 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:18.015117 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:18.015481 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:18.016695 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:18.017681 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:18.018917 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:18.019412 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:18.019595 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:18.022874 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:18.022879 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:18.022892 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:18.022922 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:18.024860 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:18.025388 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:18.025393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:18.025397 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:18.025525 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:18.025923 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:18.025927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:18.025928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:18.026334 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:18.026478 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:18.027041 (+ 563us) server_negotiation.cc:299] Negotiation successful
0806 02:48:18.027325 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":93,"threads_started":1}
I20260806 02:48:18.027521 24578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:18.013027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:37461 (local address 127.1.183.129:48901)
0806 02:48:18.013372 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:18.013387 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:18.013508 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:18.015989 (+ 2481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:18.015991 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:18.016221 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:18.016509 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:18.016527 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:18.017851 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:18.017854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:18.018786 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:18.018796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:18.019537 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:18.019541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:18.019808 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:18.020553 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:18.020569 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:18.022281 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:18.024785 (+ 2504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:18.024794 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:18.024813 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:18.025225 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:18.025608 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:18.025613 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:18.025615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:18.025783 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:18.026478 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:18.026481 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:18.026702 (+ 221us) client_negotiation.cc:769] Sending connection context
0806 02:48:18.026907 (+ 205us) client_negotiation.cc:240] Negotiation successful
0806 02:48:18.027363 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":263,"spinlock_wait_cycles":25984,"thread_start_us":62,"threads_started":1}
W20260806 02:48:18.030251 24022 leader_election.cc:341] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48b3dfeef0cd4ba4b711d09bc2410f4a (127.1.183.131:45667): Not found: Tablet not found: d7d14531c9de4d2484ef72dd2abea67f
W20260806 02:48:18.030431 24025 leader_election.cc:341] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5d733086cd7249c08b869c31d708b865 (127.1.183.130:37461): Not found: Tablet not found: d7d14531c9de4d2484ef72dd2abea67f
I20260806 02:48:18.030535 24025 leader_election.cc:302] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebe3e718f749433d879f26d19ec8e977; no voters: 48b3dfeef0cd4ba4b711d09bc2410f4a, 5d733086cd7249c08b869c31d708b865
I20260806 02:48:18.030810 24574 raft_consensus.cc:2726] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:48:18.253837 24578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:18.253507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.129:54469)
0806 02:48:18.253678 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:18.253743 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260806 02:48:18.424453 24581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:18.423780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.130:41561)
0806 02:48:18.424140 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:18.424311 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":107,"threads_started":1}
I20260806 02:48:18.623731 24582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:18.623315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.131:55095)
0806 02:48:18.623577 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:18.623635 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":78,"threads_started":1}
I20260806 02:48:19.255200 24583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:19.254752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.129:50863)
0806 02:48:19.255031 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:19.255098 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":87,"threads_started":1}
I20260806 02:48:19.425941 24584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:19.425436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.130:60437)
0806 02:48:19.425742 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:19.425817 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":86,"threads_started":1}
I20260806 02:48:19.625273 24585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:19.624753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.131:50407)
0806 02:48:19.625086 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:19.625152 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":107,"threads_started":1}
I20260806 02:48:19.657825 24587 raft_consensus.cc:482] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:48:19.657974 24587 raft_consensus.cc:504] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45667 } } peers { permanent_uuid: "5d733086cd7249c08b869c31d708b865" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37461 } } peers { permanent_uuid: "ebe3e718f749433d879f26d19ec8e977" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40057 } }
I20260806 02:48:19.658551 24587 leader_election.cc:288] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 48b3dfeef0cd4ba4b711d09bc2410f4a (127.1.183.131:45667), 5d733086cd7249c08b869c31d708b865 (127.1.183.130:37461)
W20260806 02:48:19.659427 24025 leader_election.cc:341] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5d733086cd7249c08b869c31d708b865 (127.1.183.130:37461): Not found: Tablet not found: d7d14531c9de4d2484ef72dd2abea67f
W20260806 02:48:19.659536 24022 leader_election.cc:341] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48b3dfeef0cd4ba4b711d09bc2410f4a (127.1.183.131:45667): Not found: Tablet not found: d7d14531c9de4d2484ef72dd2abea67f
I20260806 02:48:19.659603 24022 leader_election.cc:302] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebe3e718f749433d879f26d19ec8e977; no voters: 48b3dfeef0cd4ba4b711d09bc2410f4a, 5d733086cd7249c08b869c31d708b865
I20260806 02:48:19.659747 24587 raft_consensus.cc:2726] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:48:20.256561 24588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:20.256017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.129:39047)
0806 02:48:20.256389 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:20.256456 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":81,"threads_started":1}
I20260806 02:48:20.427392 24589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:20.426920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.130:42173)
0806 02:48:20.427221 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:20.427288 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":102,"threads_started":1}
I20260806 02:48:20.626756 24591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:20.626268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.131:52995)
0806 02:48:20.626582 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:20.626647 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":106,"threads_started":1}
I20260806 02:48:21.258133 24592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:21.257618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.129:38299)
0806 02:48:21.257962 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:21.258033 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":86,"threads_started":1}
I20260806 02:48:21.390827 24593 raft_consensus.cc:482] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:48:21.391032 24593 raft_consensus.cc:504] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45667 } } peers { permanent_uuid: "5d733086cd7249c08b869c31d708b865" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37461 } } peers { permanent_uuid: "ebe3e718f749433d879f26d19ec8e977" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40057 } }
I20260806 02:48:21.391542 24593 leader_election.cc:288] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 48b3dfeef0cd4ba4b711d09bc2410f4a (127.1.183.131:45667), 5d733086cd7249c08b869c31d708b865 (127.1.183.130:37461)
W20260806 02:48:21.392516 24025 leader_election.cc:341] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5d733086cd7249c08b869c31d708b865 (127.1.183.130:37461): Not found: Tablet not found: d7d14531c9de4d2484ef72dd2abea67f
W20260806 02:48:21.392671 24022 leader_election.cc:341] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48b3dfeef0cd4ba4b711d09bc2410f4a (127.1.183.131:45667): Not found: Tablet not found: d7d14531c9de4d2484ef72dd2abea67f
I20260806 02:48:21.392789 24022 leader_election.cc:302] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebe3e718f749433d879f26d19ec8e977; no voters: 48b3dfeef0cd4ba4b711d09bc2410f4a, 5d733086cd7249c08b869c31d708b865
I20260806 02:48:21.392963 24593 raft_consensus.cc:2726] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:48:21.428792 24594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:21.428325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.130:53319)
0806 02:48:21.428629 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:21.428690 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":123,"threads_started":1}
I20260806 02:48:21.629896 24595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:21.629401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.131:52275)
0806 02:48:21.629668 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:21.629747 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":95,"threads_started":1}
I20260806 02:48:22.259413 24596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:22.258985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.129:48117)
0806 02:48:22.259245 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:22.259308 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":81,"threads_started":1}
I20260806 02:48:22.430151 24597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:22.429692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.130:39037)
0806 02:48:22.429974 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:22.430046 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":98,"threads_started":1}
I20260806 02:48:22.631407 24598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:22.630894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.131:42963)
0806 02:48:22.631229 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:22.631301 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":106,"threads_started":1}
I20260806 02:48:23.260839 24600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:23.260361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.129:53183)
0806 02:48:23.260664 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:23.260731 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":96,"threads_started":1}
I20260806 02:48:23.431463 24601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:23.430923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.130:44655)
0806 02:48:23.431287 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:23.431354 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":96,"threads_started":1}
I20260806 02:48:23.632957 24603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:23.632467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.131:42823)
0806 02:48:23.632774 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:23.632837 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":74,"threads_started":1}
I20260806 02:48:24.262112 24604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:24.261619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.129:50113)
0806 02:48:24.261951 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:24.262012 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":101,"threads_started":1}
I20260806 02:48:24.433009 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:24.432461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.130:56677)
0806 02:48:24.432819 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:24.432901 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":121,"threads_started":1}
I20260806 02:48:24.634353 24607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:24.633896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.131:42479)
0806 02:48:24.634182 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:24.634247 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":81,"threads_started":1}
I20260806 02:48:25.263520 24608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:25.263038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.129:56763)
0806 02:48:25.263344 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:25.263412 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":86,"threads_started":1}
I20260806 02:48:25.434418 24609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:25.433965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.130:37733)
0806 02:48:25.434248 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:25.434313 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":91,"threads_started":1}
I20260806 02:48:25.635540 24610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:25.635113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.131:42849)
0806 02:48:25.635371 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:25.635440 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":80,"threads_started":1}
I20260806 02:48:26.160861 24611 raft_consensus.cc:482] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:48:26.161058 24611 raft_consensus.cc:504] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b3dfeef0cd4ba4b711d09bc2410f4a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45667 } } peers { permanent_uuid: "5d733086cd7249c08b869c31d708b865" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37461 } } peers { permanent_uuid: "ebe3e718f749433d879f26d19ec8e977" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40057 } }
I20260806 02:48:26.161515 24611 leader_election.cc:288] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 48b3dfeef0cd4ba4b711d09bc2410f4a (127.1.183.131:45667), 5d733086cd7249c08b869c31d708b865 (127.1.183.130:37461)
W20260806 02:48:26.162364 24025 leader_election.cc:341] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5d733086cd7249c08b869c31d708b865 (127.1.183.130:37461): Not found: Tablet not found: d7d14531c9de4d2484ef72dd2abea67f
W20260806 02:48:26.162549 24022 leader_election.cc:341] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 48b3dfeef0cd4ba4b711d09bc2410f4a (127.1.183.131:45667): Not found: Tablet not found: d7d14531c9de4d2484ef72dd2abea67f
I20260806 02:48:26.162632 24022 leader_election.cc:302] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebe3e718f749433d879f26d19ec8e977; no voters: 48b3dfeef0cd4ba4b711d09bc2410f4a, 5d733086cd7249c08b869c31d708b865
I20260806 02:48:26.162788 24611 raft_consensus.cc:2726] T d7d14531c9de4d2484ef72dd2abea67f P ebe3e718f749433d879f26d19ec8e977 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:48:26.265103 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:26.264630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.129:54907)
0806 02:48:26.264935 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:26.265003 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":102,"threads_started":1}
I20260806 02:48:26.435917 24614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:26.435309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.130:53391)
0806 02:48:26.435714 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:26.435776 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":114,"threads_started":1}
I20260806 02:48:26.637017 24615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:26.636567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.131:55147)
0806 02:48:26.636842 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:26.636913 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":81,"threads_started":1}
I20260806 02:48:27.266332 24616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:27.265920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.129:50547)
0806 02:48:27.266176 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:27.266231 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":82,"threads_started":1}
I20260806 02:48:27.437251 24618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:27.436810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.130:59519)
0806 02:48:27.437072 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:27.437138 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":78,"threads_started":1}
I20260806 02:48:27.638556 24619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:27.638075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.131:54909)
0806 02:48:27.638369 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:27.638438 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":83,"threads_started":1}
I20260806 02:48:28.267577 24622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:28.267136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.129:39495)
0806 02:48:28.267407 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:28.267476 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":78,"threads_started":1}
I20260806 02:48:28.438767 24625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:28.438223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.130:38837)
0806 02:48:28.438564 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:28.438630 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":101,"threads_started":1}
I20260806 02:48:28.640125 24626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:28.639536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.131:35705)
0806 02:48:28.639910 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:28.640017 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":107,"threads_started":1}
I20260806 02:48:29.268863 24627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:29.268429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.129:34313)
0806 02:48:29.268698 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:29.268759 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":82,"threads_started":1}
I20260806 02:48:29.440301 24630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:29.439728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.130:53571)
0806 02:48:29.440053 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:29.440124 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":85,"threads_started":1}
I20260806 02:48:29.641716 24631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:29.641193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.131:43321)
0806 02:48:29.641539 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:29.641610 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":105,"threads_started":1}
I20260806 02:48:30.270273 24632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:30.269795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.129:37903)
0806 02:48:30.270103 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:30.270170 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":109,"threads_started":1}
I20260806 02:48:30.441673 24633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:30.441184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.130:53833)
0806 02:48:30.441496 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:30.441568 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":82,"threads_started":1}
I20260806 02:48:30.643229 24634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:30.642735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.131:57745)
0806 02:48:30.643061 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:30.643126 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":110,"threads_started":1}
I20260806 02:48:31.272003 24636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:31.271564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.129:57901)
0806 02:48:31.271836 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:31.271899 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":80,"threads_started":1}
I20260806 02:48:31.443234 24637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:31.442760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.130:35137)
0806 02:48:31.443062 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:31.443129 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":82,"threads_started":1}
I20260806 02:48:31.622488 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 23946
I20260806 02:48:31.635653 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 24189
I20260806 02:48:31.649773 24639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:31.649231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46301 (local address 127.1.183.131:51855)
0806 02:48:31.649559 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:31.649642 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":107,"threads_started":1}
I20260806 02:48:31.655882 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 24386
2026-08-06T02:48:31Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (20214 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:48:31 dist-test-slave-gmjp krb5kdc[24642](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:48:31 dist-test-slave-gmjp krb5kdc[24642](info): set up 2 sockets
Aug 06 02:48:31 dist-test-slave-gmjp krb5kdc[24642](info): commencing operation
krb5kdc: starting...
W20260806 02:48:32.930240 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:48:33 dist-test-slave-gmjp krb5kdc[24642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:48:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:48:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
I20260806 02:48:33.071269 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:34847
--webserver_interface=127.1.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:45809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:34847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:48:33.193336 24658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:33.193643 24658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:33.193686 24658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:33.197924 24658 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 02:48:33.197990 24658 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 02:48:33.198006 24658 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 02:48:33.198021 24658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:48:33.198040 24658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:33.198060 24658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:33.198076 24658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:48:33.198098 24658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:48:33.203912 24658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:45809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:34847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:34847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:48:33.205154 24658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:33.206199 24658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:33.215900 24663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:33.215865 24664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:33.215963 24658 server_base.cc:1034] running on GCE node
W20260806 02:48:33.215942 24668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:33.216805 24658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:33.217998 24658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:33.219174 24658 hybrid_clock.cc:648] HybridClock initialized: now 1785984513219157 us; error 34 us; skew 500 ppm
Aug 06 02:48:33 dist-test-slave-gmjp krb5kdc[24642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984513, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:33.222256 24658 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:48:33.223223 24658 webserver.cc:460] Webserver started at http://127.1.183.190:44161/ using document root <none> and password file <none>
I20260806 02:48:33.223824 24658 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:33.223918 24658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:33.224136 24658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:48:33.226336 24658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "3f14544cf1a246e7a63bdd29f16bec3d"
format_stamp: "Formatted at 2026-08-06 02:48:33 on dist-test-slave-gmjp"
server_key: "c7e94d71f5c498643f6a4bfe7ae67808"
server_key_iv: "f377a8d6ab6f19e5dcfdab60e8128a8e"
server_key_version: "encryptionkey@0"
I20260806 02:48:33.226956 24658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "3f14544cf1a246e7a63bdd29f16bec3d"
format_stamp: "Formatted at 2026-08-06 02:48:33 on dist-test-slave-gmjp"
server_key: "c7e94d71f5c498643f6a4bfe7ae67808"
server_key_iv: "f377a8d6ab6f19e5dcfdab60e8128a8e"
server_key_version: "encryptionkey@0"
I20260806 02:48:33.231043 24658 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:48:33.233819 24680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:33.235196 24658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:48:33.235373 24658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "3f14544cf1a246e7a63bdd29f16bec3d"
format_stamp: "Formatted at 2026-08-06 02:48:33 on dist-test-slave-gmjp"
server_key: "c7e94d71f5c498643f6a4bfe7ae67808"
server_key_iv: "f377a8d6ab6f19e5dcfdab60e8128a8e"
server_key_version: "encryptionkey@0"
I20260806 02:48:33.235500 24658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:33.257405 24658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:33.260990 24658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:33.261242 24658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:33.269073 24658 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:34847
I20260806 02:48:33.269450 24754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:34847 every 8 connection(s)
I20260806 02:48:33.270205 24658 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:48:33.273375 24755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:48:33.279517 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 24658
I20260806 02:48:33.279578 24755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d: Bootstrap starting.
I20260806 02:48:33.279695 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:48:33.280050 1758 external_mini_cluster.cc:1428] Setting key edc3675bdfeeb24e154061d450cc5222
I20260806 02:48:33.282590 24755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:33.283442 24755 log.cc:824] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:33.285816 24755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d: No bootstrap required, opened a new log
Aug 06 02:48:33 dist-test-slave-gmjp krb5kdc[24642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:48:33.290755 24755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f14544cf1a246e7a63bdd29f16bec3d" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34847 } }
I20260806 02:48:33.291090 24755 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:33.291157 24755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f14544cf1a246e7a63bdd29f16bec3d, State: Initialized, Role: FOLLOWER
I20260806 02:48:33.291805 24755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f14544cf1a246e7a63bdd29f16bec3d" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34847 } }
I20260806 02:48:33.292009 24755 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:48:33.292097 24755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:48:33.292227 24755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:33.293629 24755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f14544cf1a246e7a63bdd29f16bec3d" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34847 } }
I20260806 02:48:33.294070 24755 leader_election.cc:302] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f14544cf1a246e7a63bdd29f16bec3d; no voters:
I20260806 02:48:33.294518 24755 leader_election.cc:288] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:48:33.294603 24761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:48:33.294835 24761 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [term 1 LEADER]: Becoming Leader. State: Replica: 3f14544cf1a246e7a63bdd29f16bec3d, State: Running, Role: LEADER
I20260806 02:48:33.295248 24761 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f14544cf1a246e7a63bdd29f16bec3d" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34847 } }
I20260806 02:48:33.296031 24755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:48:33.297199 24763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f14544cf1a246e7a63bdd29f16bec3d. Latest consensus state: current_term: 1 leader_uuid: "3f14544cf1a246e7a63bdd29f16bec3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f14544cf1a246e7a63bdd29f16bec3d" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34847 } } }
I20260806 02:48:33.297324 24762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f14544cf1a246e7a63bdd29f16bec3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f14544cf1a246e7a63bdd29f16bec3d" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34847 } } }
I20260806 02:48:33.297377 24763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [sys.catalog]: This master's current role is: LEADER
I20260806 02:48:33.297392 24762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [sys.catalog]: This master's current role is: LEADER
I20260806 02:48:33.297850 24780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:48:33.298226 24759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:33.281600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37090 (local address 127.1.183.190:34847)
0806 02:48:33.282121 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:33.282137 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:33.282176 (+ 39us) server_negotiation.cc:407] Connection header received
0806 02:48:33.282986 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:33.283007 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:33.283394 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:33.283780 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:33.284761 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:33.285798 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:33.287047 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:33.287570 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:33.287724 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:33.291004 (+ 3280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:33.291011 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:33.291032 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:33.291076 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:33.293446 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:33.294061 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:33.294067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:33.294074 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:33.294231 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:33.294626 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:33.294630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:33.294632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:33.295277 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:33.296160 (+ 883us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:33.296369 (+ 209us) server_negotiation.cc:299] Negotiation successful
0806 02:48:33.296730 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":173,"threads_started":1}
I20260806 02:48:33.301640 24780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:48:33.309892 24780 catalog_manager.cc:1347] Generated new cluster ID: 066d508fdb7b43a180b5e3ae9155168f
I20260806 02:48:33.309994 24780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:48:33.325903 24780 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:48:33.327486 24780 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:48:33.336117 24780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d: Generated new TSK 0
I20260806 02:48:33.336977 24780 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
I20260806 02:48:33.379467 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.129:0
--local_ip_for_outbound_sockets=127.1.183.129
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:34847
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:45809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:48:33.503077 24801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:33.503388 24801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:33.503434 24801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:33.507758 24801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:33.507838 24801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:33.507902 24801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.129
I20260806 02:48:33.513852 24801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:45809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:34847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.129
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:48:33.515034 24801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:33.516153 24801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:33.526727 24813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:33.526764 24810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:33.526834 24811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:33.527079 24801 server_base.cc:1034] running on GCE node
I20260806 02:48:33.527523 24801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:33.528321 24801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:33.529532 24801 hybrid_clock.cc:648] HybridClock initialized: now 1785984513529514 us; error 28 us; skew 500 ppm
Aug 06 02:48:33 dist-test-slave-gmjp krb5kdc[24642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984513, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:33.532580 24801 init.cc:377] Logged in from keytab as kudu/127.1.183.129@KRBTEST.COM (short username kudu)
I20260806 02:48:33.533501 24801 webserver.cc:460] Webserver started at http://127.1.183.129:39921/ using document root <none> and password file <none>
I20260806 02:48:33.534129 24801 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:33.534193 24801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:33.534427 24801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:48:33.536671 24801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/data/instance:
uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b"
format_stamp: "Formatted at 2026-08-06 02:48:33 on dist-test-slave-gmjp"
server_key: "a6c81690ae02b271667092f1e57e6cd0"
server_key_iv: "fac9fe9d154cb5733cde54b2b84eca91"
server_key_version: "encryptionkey@0"
I20260806 02:48:33.537277 24801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance:
uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b"
format_stamp: "Formatted at 2026-08-06 02:48:33 on dist-test-slave-gmjp"
server_key: "a6c81690ae02b271667092f1e57e6cd0"
server_key_iv: "fac9fe9d154cb5733cde54b2b84eca91"
server_key_version: "encryptionkey@0"
I20260806 02:48:33.541546 24801 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:48:33.544255 24821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:33.545640 24801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:48:33.545761 24801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/wal
uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b"
format_stamp: "Formatted at 2026-08-06 02:48:33 on dist-test-slave-gmjp"
server_key: "a6c81690ae02b271667092f1e57e6cd0"
server_key_iv: "fac9fe9d154cb5733cde54b2b84eca91"
server_key_version: "encryptionkey@0"
I20260806 02:48:33.545925 24801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:33.563503 24801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:33.567260 24801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:33.567441 24801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:33.568122 24801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:48:33.569108 24801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:48:33.569159 24801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:33.569233 24801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:48:33.569265 24801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:33.579118 24801 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.129:46301
I20260806 02:48:33.579211 24982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.129:46301 every 8 connection(s)
I20260806 02:48:33.580127 24801 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
I20260806 02:48:33.588200 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 24801
I20260806 02:48:33.588315 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance
I20260806 02:48:33.588640 1758 external_mini_cluster.cc:1428] Setting key 8ce23cba8428985b4c5ab8dbcf5446fa
Aug 06 02:48:33 dist-test-slave-gmjp krb5kdc[24642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984513, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:48:33.595111 24759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:33.582127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:47651 (local address 127.1.183.190:34847)
0806 02:48:33.582282 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:33.582285 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:33.583370 (+ 1085us) server_negotiation.cc:407] Connection header received
0806 02:48:33.584380 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:33.584384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:33.584462 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:33.584549 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:33.585777 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:33.586329 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:33.587505 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:33.587758 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:33.587872 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:33.590949 (+ 3077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:33.590953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:33.590956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:33.590988 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:33.592582 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:33.593149 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:33.593153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:33.593154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:33.593241 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:33.593682 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:33.593685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:33.593687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:33.594003 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:33.594118 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:33.594802 (+ 684us) server_negotiation.cc:299] Negotiation successful
0806 02:48:33.594937 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:48:33.596345 24986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:33.582524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.129:47651)
0806 02:48:33.583213 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:33.583256 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:33.584106 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:33.584741 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:33.584748 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:33.585128 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:33.585611 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:33.585624 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:33.586474 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:33.586484 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:33.587371 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:33.587380 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:33.587928 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:33.587932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:33.588107 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:33.588868 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:33.588885 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:33.590764 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:33.592711 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:33.592718 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:33.592738 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:33.593023 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:33.593352 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:33.593355 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:33.593357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:33.593566 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:33.594197 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:33.594205 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:33.594547 (+ 342us) client_negotiation.cc:769] Sending connection context
0806 02:48:33.594760 (+ 213us) client_negotiation.cc:240] Negotiation successful
0806 02:48:33.595131 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":121,"threads_started":1}
I20260806 02:48:33.598073 24983 heartbeater.cc:347] Connected to a master server at 127.1.183.190:34847
I20260806 02:48:33.598354 24983 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:33.598912 24983 heartbeater.cc:511] Master 127.1.183.190:34847 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.183.130@KRBTEST.COM; defaulting to no policy
I20260806 02:48:33.600759 24706 ts_manager.cc:194] Registered new tserver with Master: d6dcb23d9e4f4ab49608bacef00f2a4b (127.1.183.129:46301)
I20260806 02:48:33.601984 24706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.129@KRBTEST.COM'} at 127.1.183.129:47651
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
I20260806 02:48:33.621423 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.130:0
--local_ip_for_outbound_sockets=127.1.183.130
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:34847
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:45809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:48:33.741227 24993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:33.741508 24993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:33.741582 24993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:33.745882 24993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:33.745954 24993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:33.746047 24993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.130
I20260806 02:48:33.751859 24993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:45809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:34847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.130
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:48:33.753050 24993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:33.754136 24993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:33.764731 25004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:33.764715 25000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:33.764731 24999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:33.765196 24993 server_base.cc:1034] running on GCE node
I20260806 02:48:33.765700 24993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:33.766486 24993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:33.767705 24993 hybrid_clock.cc:648] HybridClock initialized: now 1785984513767680 us; error 46 us; skew 500 ppm
Aug 06 02:48:33 dist-test-slave-gmjp krb5kdc[24642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984513, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:33.770990 24993 init.cc:377] Logged in from keytab as kudu/127.1.183.130@KRBTEST.COM (short username kudu)
I20260806 02:48:33.771967 24993 webserver.cc:460] Webserver started at http://127.1.183.130:46265/ using document root <none> and password file <none>
I20260806 02:48:33.772632 24993 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:33.772703 24993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:33.772886 24993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:48:33.775087 24993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/data/instance:
uuid: "984378e530c349159d547c20dfef131f"
format_stamp: "Formatted at 2026-08-06 02:48:33 on dist-test-slave-gmjp"
server_key: "80db4ce4d4cd2f3544e1f74b38c73a38"
server_key_iv: "c0558eb2182cb3d003d28da14c166b42"
server_key_version: "encryptionkey@0"
I20260806 02:48:33.775650 24993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance:
uuid: "984378e530c349159d547c20dfef131f"
format_stamp: "Formatted at 2026-08-06 02:48:33 on dist-test-slave-gmjp"
server_key: "80db4ce4d4cd2f3544e1f74b38c73a38"
server_key_iv: "c0558eb2182cb3d003d28da14c166b42"
server_key_version: "encryptionkey@0"
I20260806 02:48:33.779839 24993 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:48:33.782732 25012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:33.784303 24993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:48:33.784422 24993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/wal
uuid: "984378e530c349159d547c20dfef131f"
format_stamp: "Formatted at 2026-08-06 02:48:33 on dist-test-slave-gmjp"
server_key: "80db4ce4d4cd2f3544e1f74b38c73a38"
server_key_iv: "c0558eb2182cb3d003d28da14c166b42"
server_key_version: "encryptionkey@0"
I20260806 02:48:33.784533 24993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:33.797824 24993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:33.801280 24993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:33.801512 24993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:33.802194 24993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:48:33.803128 24993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:48:33.803203 24993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:33.803277 24993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:48:33.803332 24993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:33.813102 24993 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.130:45295
I20260806 02:48:33.813133 25179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.130:45295 every 8 connection(s)
I20260806 02:48:33.814126 24993 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
I20260806 02:48:33.820057 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 24993
I20260806 02:48:33.820238 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance
I20260806 02:48:33.820585 1758 external_mini_cluster.cc:1428] Setting key aaf166cefee7051f6ecbdd6112ed1012
Aug 06 02:48:33 dist-test-slave-gmjp krb5kdc[24642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984513, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:48:33.829207 24759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:33.816092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:58853 (local address 127.1.183.190:34847)
0806 02:48:33.816264 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:33.816268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:33.817468 (+ 1200us) server_negotiation.cc:407] Connection header received
0806 02:48:33.818423 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:33.818426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:33.818503 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:33.818611 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:33.819977 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:33.820802 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:33.822088 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:33.822392 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:33.822511 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:33.825305 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:33.825308 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:33.825310 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:33.825337 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:33.826837 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:33.827434 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:33.827439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:33.827447 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:33.827527 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:33.827892 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:33.827896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:33.827898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:33.828114 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:33.828302 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:33.828876 (+ 574us) server_negotiation.cc:299] Negotiation successful
0806 02:48:33.829033 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 02:48:33.830133 25183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:33.816569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.130:58853)
0806 02:48:33.817292 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:33.817345 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:33.818190 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:33.818800 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:33.818807 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:33.819192 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:33.819772 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:33.819796 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:33.820953 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:33.820957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:33.821933 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:33.821945 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:33.822573 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:33.822577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:33.822741 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:33.823436 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:33.823454 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:33.825093 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:33.826981 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:33.826988 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:33.827008 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:33.827310 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:33.827626 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:33.827630 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:33.827632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:33.827789 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:33.828302 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:33.828311 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:33.828618 (+ 307us) client_negotiation.cc:769] Sending connection context
0806 02:48:33.828830 (+ 212us) client_negotiation.cc:240] Negotiation successful
0806 02:48:33.829210 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":143,"threads_started":1}
WARNING: no policy specified for kudu/127.1.183.131@KRBTEST.COM; defaulting to no policy
I20260806 02:48:33.831565 25181 heartbeater.cc:347] Connected to a master server at 127.1.183.190:34847
I20260806 02:48:33.831877 25181 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:33.832552 25181 heartbeater.cc:511] Master 127.1.183.190:34847 requested a full tablet report, sending...
I20260806 02:48:33.833751 24706 ts_manager.cc:194] Registered new tserver with Master: 984378e530c349159d547c20dfef131f (127.1.183.130:45295)
I20260806 02:48:33.834393 24706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.130@KRBTEST.COM'} at 127.1.183.130:58853
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
I20260806 02:48:33.853638 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.131:0
--local_ip_for_outbound_sockets=127.1.183.131
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:34847
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:45809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:48:33.973897 25190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:33.974215 25190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:33.974259 25190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:33.978399 25190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:33.978492 25190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:33.978556 25190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.131
I20260806 02:48:33.984267 25190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:45809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:34847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.131
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:48:33.985447 25190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:33.986516 25190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:33.997174 25197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:33.997215 25196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:33.997414 25200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:33.997720 25190 server_base.cc:1034] running on GCE node
I20260806 02:48:33.998240 25190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:33.998991 25190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:34.000198 25190 hybrid_clock.cc:648] HybridClock initialized: now 1785984514000138 us; error 32 us; skew 500 ppm
Aug 06 02:48:33 dist-test-slave-gmjp krb5kdc[24642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984513, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:34.003136 25190 init.cc:377] Logged in from keytab as kudu/127.1.183.131@KRBTEST.COM (short username kudu)
I20260806 02:48:34.004104 25190 webserver.cc:460] Webserver started at http://127.1.183.131:33283/ using document root <none> and password file <none>
I20260806 02:48:34.004748 25190 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:34.004849 25190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:34.005097 25190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:48:34.007344 25190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/data/instance:
uuid: "aad0f35208db44fe8cf5fcc4b6a00a89"
format_stamp: "Formatted at 2026-08-06 02:48:34 on dist-test-slave-gmjp"
server_key: "c9be08837258de05c57d727320032012"
server_key_iv: "a6ce26e6205e3eec22eec21ad4c27bbe"
server_key_version: "encryptionkey@0"
I20260806 02:48:34.007974 25190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance:
uuid: "aad0f35208db44fe8cf5fcc4b6a00a89"
format_stamp: "Formatted at 2026-08-06 02:48:34 on dist-test-slave-gmjp"
server_key: "c9be08837258de05c57d727320032012"
server_key_iv: "a6ce26e6205e3eec22eec21ad4c27bbe"
server_key_version: "encryptionkey@0"
I20260806 02:48:34.012058 25190 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260806 02:48:34.014757 25211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:34.016072 25190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:48:34.016254 25190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/wal
uuid: "aad0f35208db44fe8cf5fcc4b6a00a89"
format_stamp: "Formatted at 2026-08-06 02:48:34 on dist-test-slave-gmjp"
server_key: "c9be08837258de05c57d727320032012"
server_key_iv: "a6ce26e6205e3eec22eec21ad4c27bbe"
server_key_version: "encryptionkey@0"
I20260806 02:48:34.016410 25190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:34.028404 25190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:34.031872 25190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:34.032114 25190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:34.032830 25190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:48:34.033768 25190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:48:34.033845 25190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:34.033921 25190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:48:34.033973 25190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:34.043766 25190 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.131:42193
I20260806 02:48:34.043774 25375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.131:42193 every 8 connection(s)
I20260806 02:48:34.044837 25190 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
I20260806 02:48:34.052137 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 25190
I20260806 02:48:34.052297 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance
I20260806 02:48:34.052631 1758 external_mini_cluster.cc:1428] Setting key e39422a95872f42fef5758590a290a38
Aug 06 02:48:34 dist-test-slave-gmjp krb5kdc[24642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984513, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:48:34.059809 24759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:34.046797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:32989 (local address 127.1.183.190:34847)
0806 02:48:34.046976 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:34.046982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:34.048089 (+ 1107us) server_negotiation.cc:407] Connection header received
0806 02:48:34.049054 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:34.049057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:34.049123 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:34.049208 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:34.050466 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:34.051019 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:34.052214 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:34.052580 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:34.052715 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:34.055797 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:34.055800 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:34.055803 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:34.055833 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:34.057717 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:34.058239 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:34.058242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:34.058244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:34.058341 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:34.058684 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:34.058686 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:34.058688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:34.058903 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:34.058998 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:34.059539 (+ 541us) server_negotiation.cc:299] Negotiation successful
0806 02:48:34.059679 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 02:48:34.060755 25381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:34.047248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.131:32989)
0806 02:48:34.047930 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:34.047974 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:34.048835 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:34.049356 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:34.049363 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:34.049742 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:34.050315 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:34.050332 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:34.051137 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:34.051140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:34.052040 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:34.052049 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:34.052792 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:34.052796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:34.052962 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:34.053694 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:34.053717 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:34.055587 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:34.057831 (+ 2244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:34.057836 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:34.057851 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:34.058124 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:34.058434 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:34.058436 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:34.058437 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:34.058590 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:34.059003 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:34.059008 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:34.059268 (+ 260us) client_negotiation.cc:769] Sending connection context
0806 02:48:34.059439 (+ 171us) client_negotiation.cc:240] Negotiation successful
0806 02:48:34.059755 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":121,"threads_started":1}
I20260806 02:48:34.061996 25378 heartbeater.cc:347] Connected to a master server at 127.1.183.190:34847
I20260806 02:48:34.062264 25378 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:34.062814 25378 heartbeater.cc:511] Master 127.1.183.190:34847 requested a full tablet report, sending...
I20260806 02:48:34.063933 24706 ts_manager.cc:194] Registered new tserver with Master: aad0f35208db44fe8cf5fcc4b6a00a89 (127.1.183.131:42193)
I20260806 02:48:34.064581 24706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.131@KRBTEST.COM'} at 127.1.183.131:32989
I20260806 02:48:34.067003 1758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:48:34.076671 24759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:34.069139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37096 (local address 127.1.183.190:34847)
0806 02:48:34.069301 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:34.069319 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:34.069450 (+ 131us) server_negotiation.cc:407] Connection header received
0806 02:48:34.069568 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:34.069570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:34.069610 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:34.069679 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:34.070374 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:34.070884 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:34.071673 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:34.071878 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:34.071952 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:34.073359 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:34.073362 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:34.073364 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:34.073389 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:34.074917 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:34.075301 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:34.075304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:34.075305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:34.075387 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:34.075647 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:34.075650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:34.075652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:34.075938 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:34.076061 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:34.076315 (+ 254us) server_negotiation.cc:299] Negotiation successful
0806 02:48:34.076426 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 02:48:34.080125 24706 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37096:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:48:34.084000 24706 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:48:34.100826 25396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:34.093813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:42193 (local address 127.0.0.1:37674)
0806 02:48:34.094596 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:34.094651 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:34.094795 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:34.096114 (+ 1319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:34.096116 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:34.096134 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:34.096261 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:34.096268 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:34.097645 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:34.097649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:34.098945 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:34.098957 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:34.100095 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:34.100099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:34.100272 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:34.100328 (+ 56us) client_negotiation.cc:769] Sending connection context
0806 02:48:34.100439 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 02:48:34.100604 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":111,"spinlock_wait_cycles":2688,"thread_start_us":123,"threads_started":1}
I20260806 02:48:34.100836 25402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:34.094614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33064 (local address 127.1.183.129:46301)
0806 02:48:34.095253 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:34.095284 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:34.095311 (+ 27us) server_negotiation.cc:407] Connection header received
0806 02:48:34.095397 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:34.095402 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:34.095609 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:34.095757 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:34.096254 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:34.097289 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:34.099421 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:34.100281 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:34.100395 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:34.100549 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:34.100612 (+ 63us) server_negotiation.cc:299] Negotiation successful
0806 02:48:34.100707 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":114,"threads_started":1}
I20260806 02:48:34.100849 25397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:34.093871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37778 (local address 127.1.183.130:45295)
0806 02:48:34.094399 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:34.094413 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:34.094935 (+ 522us) server_negotiation.cc:407] Connection header received
0806 02:48:34.095029 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:34.095036 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:34.095263 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:34.095442 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:34.095992 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:34.097346 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:34.099343 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:34.099919 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:34.100066 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:34.100272 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:34.100582 (+ 310us) server_negotiation.cc:299] Negotiation successful
0806 02:48:34.100696 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":165,"threads_started":1}
I20260806 02:48:34.100837 25399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:34.094095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:46301 (local address 127.0.0.1:33064)
0806 02:48:34.095085 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:34.095102 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:34.095200 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:34.095970 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:34.095974 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:34.095993 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:34.096101 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:34.096107 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:34.097405 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:34.097411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:34.099280 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:34.099291 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:34.100408 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:34.100414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:34.100501 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:34.100528 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 02:48:34.100589 (+ 61us) client_negotiation.cc:240] Negotiation successful
0806 02:48:34.100658 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":918,"mutex_wait_us":180,"thread_start_us":98,"threads_started":1}
I20260806 02:48:34.101817 25403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:34.094619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37674 (local address 127.1.183.131:42193)
0806 02:48:34.095445 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:34.095460 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:34.095484 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:48:34.095564 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:34.095699 (+ 135us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:34.095900 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:34.096124 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:34.096386 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:34.097505 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:34.099087 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:34.099944 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:34.101408 (+ 1464us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:34.101550 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:34.101593 (+ 43us) server_negotiation.cc:299] Negotiation successful
0806 02:48:34.101666 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":171,"threads_started":1}
I20260806 02:48:34.102169 25394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:34.093751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:45295 (local address 127.0.0.1:37778)
0806 02:48:34.094596 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:34.094651 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:34.094795 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:34.095650 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:34.095658 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:34.095686 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:34.095850 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:34.095858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:34.097490 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:34.097494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:34.099212 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:34.099225 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:34.100069 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:34.100073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:34.100248 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:34.100308 (+ 60us) client_negotiation.cc:769] Sending connection context
0806 02:48:34.100441 (+ 133us) client_negotiation.cc:240] Negotiation successful
0806 02:48:34.102037 (+ 1596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":1668,"spinlock_wait_cycles":7936,"thread_start_us":100,"threads_started":1}
I20260806 02:48:34.104576 24889 tablet_service.cc:1467] Processing CreateTablet for tablet 45692a76b79a40049fe460bc2fbb4736 (DEFAULT_TABLE table=test-table [id=305b440421e5442483245b9453762240]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:48:34.104697 25284 tablet_service.cc:1467] Processing CreateTablet for tablet 45692a76b79a40049fe460bc2fbb4736 (DEFAULT_TABLE table=test-table [id=305b440421e5442483245b9453762240]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:48:34.104696 25078 tablet_service.cc:1467] Processing CreateTablet for tablet 45692a76b79a40049fe460bc2fbb4736 (DEFAULT_TABLE table=test-table [id=305b440421e5442483245b9453762240]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:48:34.105700 24889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45692a76b79a40049fe460bc2fbb4736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:48:34.105782 25078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45692a76b79a40049fe460bc2fbb4736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:48:34.105803 25284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45692a76b79a40049fe460bc2fbb4736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:48:34.113196 25408 tablet_bootstrap.cc:492] T 45692a76b79a40049fe460bc2fbb4736 P 984378e530c349159d547c20dfef131f: Bootstrap starting.
I20260806 02:48:34.115659 25408 tablet_bootstrap.cc:654] T 45692a76b79a40049fe460bc2fbb4736 P 984378e530c349159d547c20dfef131f: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:34.116503 25408 log.cc:824] T 45692a76b79a40049fe460bc2fbb4736 P 984378e530c349159d547c20dfef131f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:34.116528 25405 tablet_bootstrap.cc:492] T 45692a76b79a40049fe460bc2fbb4736 P d6dcb23d9e4f4ab49608bacef00f2a4b: Bootstrap starting.
I20260806 02:48:34.117288 25406 tablet_bootstrap.cc:492] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89: Bootstrap starting.
I20260806 02:48:34.118773 25405 tablet_bootstrap.cc:654] T 45692a76b79a40049fe460bc2fbb4736 P d6dcb23d9e4f4ab49608bacef00f2a4b: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:34.118906 25408 tablet_bootstrap.cc:492] T 45692a76b79a40049fe460bc2fbb4736 P 984378e530c349159d547c20dfef131f: No bootstrap required, opened a new log
I20260806 02:48:34.119128 25408 ts_tablet_manager.cc:1397] T 45692a76b79a40049fe460bc2fbb4736 P 984378e530c349159d547c20dfef131f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:48:34.120185 25406 tablet_bootstrap.cc:654] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:34.120458 25405 log.cc:824] T 45692a76b79a40049fe460bc2fbb4736 P d6dcb23d9e4f4ab49608bacef00f2a4b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:34.121146 25406 log.cc:824] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:34.123128 25405 tablet_bootstrap.cc:492] T 45692a76b79a40049fe460bc2fbb4736 P d6dcb23d9e4f4ab49608bacef00f2a4b: No bootstrap required, opened a new log
I20260806 02:48:34.123395 25405 ts_tablet_manager.cc:1397] T 45692a76b79a40049fe460bc2fbb4736 P d6dcb23d9e4f4ab49608bacef00f2a4b: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260806 02:48:34.124231 25408 raft_consensus.cc:348] T 45692a76b79a40049fe460bc2fbb4736 P 984378e530c349159d547c20dfef131f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984378e530c349159d547c20dfef131f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 45295 } } peers { permanent_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 42193 } } peers { permanent_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46301 } }
I20260806 02:48:34.124555 25408 raft_consensus.cc:374] T 45692a76b79a40049fe460bc2fbb4736 P 984378e530c349159d547c20dfef131f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:34.124614 25408 raft_consensus.cc:729] T 45692a76b79a40049fe460bc2fbb4736 P 984378e530c349159d547c20dfef131f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 984378e530c349159d547c20dfef131f, State: Initialized, Role: FOLLOWER
I20260806 02:48:34.124831 25406 tablet_bootstrap.cc:492] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89: No bootstrap required, opened a new log
I20260806 02:48:34.125025 25406 ts_tablet_manager.cc:1397] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260806 02:48:34.125099 25408 consensus_queue.cc:260] T 45692a76b79a40049fe460bc2fbb4736 P 984378e530c349159d547c20dfef131f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984378e530c349159d547c20dfef131f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 45295 } } peers { permanent_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 42193 } } peers { permanent_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46301 } }
I20260806 02:48:34.125844 25181 heartbeater.cc:503] Master 127.1.183.190:34847 was elected leader, sending a full tablet report...
I20260806 02:48:34.127353 25408 ts_tablet_manager.cc:1428] T 45692a76b79a40049fe460bc2fbb4736 P 984378e530c349159d547c20dfef131f: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260806 02:48:34.127617 25405 raft_consensus.cc:348] T 45692a76b79a40049fe460bc2fbb4736 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984378e530c349159d547c20dfef131f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 45295 } } peers { permanent_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 42193 } } peers { permanent_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46301 } }
I20260806 02:48:34.127945 25405 raft_consensus.cc:374] T 45692a76b79a40049fe460bc2fbb4736 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:34.128029 25405 raft_consensus.cc:729] T 45692a76b79a40049fe460bc2fbb4736 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6dcb23d9e4f4ab49608bacef00f2a4b, State: Initialized, Role: FOLLOWER
I20260806 02:48:34.128576 25405 consensus_queue.cc:260] T 45692a76b79a40049fe460bc2fbb4736 P d6dcb23d9e4f4ab49608bacef00f2a4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984378e530c349159d547c20dfef131f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 45295 } } peers { permanent_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 42193 } } peers { permanent_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46301 } }
I20260806 02:48:34.128952 25406 raft_consensus.cc:348] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984378e530c349159d547c20dfef131f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 45295 } } peers { permanent_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 42193 } } peers { permanent_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46301 } }
I20260806 02:48:34.129329 25406 raft_consensus.cc:374] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:34.129432 25406 raft_consensus.cc:729] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad0f35208db44fe8cf5fcc4b6a00a89, State: Initialized, Role: FOLLOWER
I20260806 02:48:34.129456 24983 heartbeater.cc:503] Master 127.1.183.190:34847 was elected leader, sending a full tablet report...
I20260806 02:48:34.129745 25405 ts_tablet_manager.cc:1428] T 45692a76b79a40049fe460bc2fbb4736 P d6dcb23d9e4f4ab49608bacef00f2a4b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:48:34.130007 25406 consensus_queue.cc:260] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984378e530c349159d547c20dfef131f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 45295 } } peers { permanent_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 42193 } } peers { permanent_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46301 } }
I20260806 02:48:34.131413 25406 ts_tablet_manager.cc:1428] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260806 02:48:34.131808 25378 heartbeater.cc:503] Master 127.1.183.190:34847 was elected leader, sending a full tablet report...
W20260806 02:48:34.295919 25379 tablet.cc:2369] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:48:34.315419 25182 tablet.cc:2369] T 45692a76b79a40049fe460bc2fbb4736 P 984378e530c349159d547c20dfef131f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:48:34.331709 24985 tablet.cc:2369] T 45692a76b79a40049fe460bc2fbb4736 P d6dcb23d9e4f4ab49608bacef00f2a4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:48:34.505409 25416 raft_consensus.cc:482] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:48:34.505671 25416 raft_consensus.cc:504] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984378e530c349159d547c20dfef131f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 45295 } } peers { permanent_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 42193 } } peers { permanent_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46301 } }
I20260806 02:48:34.506991 25416 leader_election.cc:288] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 984378e530c349159d547c20dfef131f (127.1.183.130:45295), d6dcb23d9e4f4ab49608bacef00f2a4b (127.1.183.129:46301)
I20260806 02:48:34.510905 25381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:34.507195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:45295 (local address 127.1.183.131:34143)
0806 02:48:34.507336 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:34.507358 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:34.507469 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:34.507857 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:34.507860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:34.507882 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:34.508006 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:34.508011 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:34.508970 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:34.508974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:34.509771 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:34.509781 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:34.510455 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:34.510459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:34.510570 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:34.510588 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 02:48:34.510674 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 02:48:34.510750 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":3}
I20260806 02:48:34.510895 25397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:34.507310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:34143 (local address 127.1.183.130:45295)
0806 02:48:34.507599 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:34.507605 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:34.507619 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:48:34.507679 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:34.507682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:34.507736 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:34.507825 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:34.508120 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:34.508852 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:34.509903 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:34.510312 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:34.510455 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:34.510511 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:34.510692 (+ 181us) server_negotiation.cc:299] Negotiation successful
0806 02:48:34.510766 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260806 02:48:34.511142 25417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:34.507303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:46301 (local address 127.1.183.131:44453)
0806 02:48:34.507735 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:34.507747 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:34.507839 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:34.508371 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:34.508374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:34.508394 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:34.508510 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:34.508516 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:34.509230 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:34.509236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:34.510033 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:34.510042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:34.510706 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:34.510710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:34.510807 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:34.510828 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 02:48:34.510923 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 02:48:34.510990 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":3,"thread_start_us":188,"threads_started":1}
I20260806 02:48:34.511126 25402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:34.507570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:44453 (local address 127.1.183.129:46301)
0806 02:48:34.507707 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:34.507710 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:34.508104 (+ 394us) server_negotiation.cc:407] Connection header received
0806 02:48:34.508181 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:34.508183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:34.508259 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:34.508327 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:34.508624 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:34.509112 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:34.510168 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:34.510575 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:34.510706 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:34.510757 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:34.510941 (+ 184us) server_negotiation.cc:299] Negotiation successful
0806 02:48:34.511016 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:48:34.511704 25111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45692a76b79a40049fe460bc2fbb4736" candidate_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "984378e530c349159d547c20dfef131f" is_pre_election: true
I20260806 02:48:34.511842 24915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45692a76b79a40049fe460bc2fbb4736" candidate_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b" is_pre_election: true
I20260806 02:48:34.512068 25111 raft_consensus.cc:2454] T 45692a76b79a40049fe460bc2fbb4736 P 984378e530c349159d547c20dfef131f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad0f35208db44fe8cf5fcc4b6a00a89 in term 0.
I20260806 02:48:34.512253 24915 raft_consensus.cc:2454] T 45692a76b79a40049fe460bc2fbb4736 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad0f35208db44fe8cf5fcc4b6a00a89 in term 0.
I20260806 02:48:34.512655 25214 leader_election.cc:302] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 984378e530c349159d547c20dfef131f, aad0f35208db44fe8cf5fcc4b6a00a89; no voters:
I20260806 02:48:34.512956 25416 raft_consensus.cc:2781] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:48:34.513051 25416 raft_consensus.cc:482] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:48:34.513095 25416 raft_consensus.cc:3037] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:34.514190 25416 raft_consensus.cc:504] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984378e530c349159d547c20dfef131f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 45295 } } peers { permanent_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 42193 } } peers { permanent_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46301 } }
I20260806 02:48:34.514570 25416 leader_election.cc:288] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89 [CANDIDATE]: Term 1 election: Requested vote from peers 984378e530c349159d547c20dfef131f (127.1.183.130:45295), d6dcb23d9e4f4ab49608bacef00f2a4b (127.1.183.129:46301)
I20260806 02:48:34.514930 25111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45692a76b79a40049fe460bc2fbb4736" candidate_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "984378e530c349159d547c20dfef131f"
I20260806 02:48:34.515010 24915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45692a76b79a40049fe460bc2fbb4736" candidate_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b"
I20260806 02:48:34.515081 25111 raft_consensus.cc:3037] T 45692a76b79a40049fe460bc2fbb4736 P 984378e530c349159d547c20dfef131f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:34.515152 24915 raft_consensus.cc:3037] T 45692a76b79a40049fe460bc2fbb4736 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:34.516211 25111 raft_consensus.cc:2454] T 45692a76b79a40049fe460bc2fbb4736 P 984378e530c349159d547c20dfef131f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad0f35208db44fe8cf5fcc4b6a00a89 in term 1.
I20260806 02:48:34.516307 24915 raft_consensus.cc:2454] T 45692a76b79a40049fe460bc2fbb4736 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad0f35208db44fe8cf5fcc4b6a00a89 in term 1.
I20260806 02:48:34.516623 25214 leader_election.cc:302] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 984378e530c349159d547c20dfef131f, aad0f35208db44fe8cf5fcc4b6a00a89; no voters:
I20260806 02:48:34.516788 25416 raft_consensus.cc:2781] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:48:34.517001 25416 raft_consensus.cc:686] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89 [term 1 LEADER]: Becoming Leader. State: Replica: aad0f35208db44fe8cf5fcc4b6a00a89, State: Running, Role: LEADER
I20260806 02:48:34.517330 25416 consensus_queue.cc:237] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984378e530c349159d547c20dfef131f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 45295 } } peers { permanent_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 42193 } } peers { permanent_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46301 } }
I20260806 02:48:34.520875 24706 catalog_manager.cc:5495] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89 reported cstate change: term changed from 0 to 1, leader changed from <none> to aad0f35208db44fe8cf5fcc4b6a00a89 (127.1.183.131). New cstate: current_term: 1 leader_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984378e530c349159d547c20dfef131f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 45295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 42193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46301 } health_report { overall_health: UNKNOWN } } }
I20260806 02:48:34.562114 25403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:34.558323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37686 (local address 127.1.183.131:42193)
0806 02:48:34.558497 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:34.558501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:34.558518 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:48:34.558602 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:34.558605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:34.558660 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:34.558770 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:48:34.559026 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:34.559565 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:34.560539 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:34.560807 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:34.560901 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:34.561176 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:48:34.561609 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:48:34.561704 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:34.561862 (+ 158us) server_negotiation.cc:299] Negotiation successful
0806 02:48:34.561924 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 02:48:34.568950 25111 raft_consensus.cc:1261] T 45692a76b79a40049fe460bc2fbb4736 P 984378e530c349159d547c20dfef131f [term 1 FOLLOWER]: Refusing update from remote peer aad0f35208db44fe8cf5fcc4b6a00a89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:48:34.568948 24915 raft_consensus.cc:1261] T 45692a76b79a40049fe460bc2fbb4736 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 1 FOLLOWER]: Refusing update from remote peer aad0f35208db44fe8cf5fcc4b6a00a89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:48:34.569761 25416 consensus_queue.cc:1035] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:48:34.569988 25418 consensus_queue.cc:1035] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "984378e530c349159d547c20dfef131f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 45295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:48:34.578531 25423 mvcc.cc:205] Tried to move back new op lower bound from 7315392571665563648 to 7315392571465859072. Current Snapshot: MvccSnapshot[applied={T|T < 7315392571665563648}]
I20260806 02:48:34.580212 25425 mvcc.cc:205] Tried to move back new op lower bound from 7315392571665563648 to 7315392571465859072. Current Snapshot: MvccSnapshot[applied={T|T < 7315392571665563648}]
I20260806 02:48:34.591482 24705 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:48:34.591693 24705 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:48:34.594408 24705 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d: Sending DeleteTablet for 3 replicas of tablet 45692a76b79a40049fe460bc2fbb4736
I20260806 02:48:34.595297 25284 tablet_service.cc:1514] Processing DeleteTablet for tablet 45692a76b79a40049fe460bc2fbb4736 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:48:34 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:37674
I20260806 02:48:34.595309 25078 tablet_service.cc:1514] Processing DeleteTablet for tablet 45692a76b79a40049fe460bc2fbb4736 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:48:34 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:37778
I20260806 02:48:34.595860 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 24801
I20260806 02:48:34.595847 24889 tablet_service.cc:1514] Processing DeleteTablet for tablet 45692a76b79a40049fe460bc2fbb4736 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:48:34 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:33064
I20260806 02:48:34.603765 25438 tablet_replica.cc:330] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89: stopping tablet replica
I20260806 02:48:34.604346 25438 raft_consensus.cc:2229] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:48:34.604882 25438 raft_consensus.cc:2258] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:48:34.606774 25438 ts_tablet_manager.cc:1905] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:48:34.608485 25437 tablet_replica.cc:330] T 45692a76b79a40049fe460bc2fbb4736 P 984378e530c349159d547c20dfef131f: stopping tablet replica
I20260806 02:48:34.608963 25437 raft_consensus.cc:2229] T 45692a76b79a40049fe460bc2fbb4736 P 984378e530c349159d547c20dfef131f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:48:34.609484 25437 raft_consensus.cc:2258] T 45692a76b79a40049fe460bc2fbb4736 P 984378e530c349159d547c20dfef131f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:48:34.610020 25438 ts_tablet_manager.cc:1918] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:48:34.610180 25438 log.cc:1196] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/wal/wals/45692a76b79a40049fe460bc2fbb4736
I20260806 02:48:34.610596 25438 ts_tablet_manager.cc:1939] T 45692a76b79a40049fe460bc2fbb4736 P aad0f35208db44fe8cf5fcc4b6a00a89: Deleting consensus metadata
I20260806 02:48:34.611481 25437 ts_tablet_manager.cc:1905] T 45692a76b79a40049fe460bc2fbb4736 P 984378e530c349159d547c20dfef131f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:48:34.612098 24686 catalog_manager.cc:4841] TS aad0f35208db44fe8cf5fcc4b6a00a89 (127.1.183.131:42193): tablet 45692a76b79a40049fe460bc2fbb4736 (table test-table [id=305b440421e5442483245b9453762240]) successfully deleted
I20260806 02:48:34.615236 25437 ts_tablet_manager.cc:1918] T 45692a76b79a40049fe460bc2fbb4736 P 984378e530c349159d547c20dfef131f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260806 02:48:34.615236 24685 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.183.129:46301 (error 108)
I20260806 02:48:34.615411 25437 log.cc:1196] T 45692a76b79a40049fe460bc2fbb4736 P 984378e530c349159d547c20dfef131f: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/wal/wals/45692a76b79a40049fe460bc2fbb4736
I20260806 02:48:34.615550 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 24993
I20260806 02:48:34.619208 25440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:34.616258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:46301 (local address 127.0.0.1:33076)
0806 02:48:34.618939 (+ 2681us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:34.619049 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.129:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2563,"thread_start_us":1593,"threads_started":1}
W20260806 02:48:34.619436 24685 catalog_manager.cc:4584] TS d6dcb23d9e4f4ab49608bacef00f2a4b (127.1.183.129:46301): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 45692a76b79a40049fe460bc2fbb4736: Network error: Client connection negotiation failed: client connection to 127.1.183.129:46301: connect: Connection refused (error 111)
I20260806 02:48:34.627326 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 25190
I20260806 02:48:34.637895 25440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:34.627631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:45295 (local address 127.0.0.1:37790)
0806 02:48:34.637651 (+ 10020us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:34.637750 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.130:45295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9851}
I20260806 02:48:34.644202 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 24658
I20260806 02:48:54.659341 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:34847
--webserver_interface=127.1.183.190
--webserver_port=44161
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:45809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:34847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:48:54.781242 25442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:54.781538 25442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:54.781579 25442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:54.785871 25442 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 02:48:54.785943 25442 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 02:48:54.785960 25442 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 02:48:54.785974 25442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:48:54.785992 25442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:54.786010 25442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:54.786026 25442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:48:54.786089 25442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:48:54.791637 25442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:45809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:34847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:34847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=44161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:48:54.792987 25442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:54.794127 25442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:54.808439 25451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:54.808439 25455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:54.808568 25442 server_base.cc:1034] running on GCE node
W20260806 02:48:54.808439 25450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:54.809505 25442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:54.810781 25442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:54.811964 25442 hybrid_clock.cc:648] HybridClock initialized: now 1785984534811944 us; error 31 us; skew 500 ppm
Aug 06 02:48:54 dist-test-slave-gmjp krb5kdc[24642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984534, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:54.815119 25442 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:48:54.816123 25442 webserver.cc:460] Webserver started at http://127.1.183.190:44161/ using document root <none> and password file <none>
I20260806 02:48:54.816790 25442 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:54.816912 25442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:54.821362 25442 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:48:54.823760 25466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:54.825258 25442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:48:54.825430 25442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "3f14544cf1a246e7a63bdd29f16bec3d"
format_stamp: "Formatted at 2026-08-06 02:48:33 on dist-test-slave-gmjp"
server_key: "c7e94d71f5c498643f6a4bfe7ae67808"
server_key_iv: "f377a8d6ab6f19e5dcfdab60e8128a8e"
server_key_version: "encryptionkey@0"
I20260806 02:48:54.826107 25442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:54.836838 25442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:54.840574 25442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:54.840826 25442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:54.848384 25442 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:34847
I20260806 02:48:54.848701 25537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:34847 every 8 connection(s)
I20260806 02:48:54.849597 25442 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:48:54.854373 25538 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:48:54.856191 25538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d: Bootstrap starting.
I20260806 02:48:54.857973 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 25442
I20260806 02:48:54.858497 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.129:46301
--local_ip_for_outbound_sockets=127.1.183.129
--tserver_master_addrs=127.1.183.190:34847
--webserver_port=39921
--webserver_interface=127.1.183.129
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:45809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:48:54.874635 25538 log.cc:824] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:54.883981 25538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:48:54.884467 25538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d: Bootstrap complete.
I20260806 02:48:54.888682 25538 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f14544cf1a246e7a63bdd29f16bec3d" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34847 } }
I20260806 02:48:54.889174 25538 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f14544cf1a246e7a63bdd29f16bec3d, State: Initialized, Role: FOLLOWER
I20260806 02:48:54.889714 25538 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f14544cf1a246e7a63bdd29f16bec3d" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34847 } }
I20260806 02:48:54.889873 25538 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:48:54.889969 25538 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:48:54.890089 25538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:48:54.892063 25538 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f14544cf1a246e7a63bdd29f16bec3d" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34847 } }
I20260806 02:48:54.892445 25538 leader_election.cc:302] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f14544cf1a246e7a63bdd29f16bec3d; no voters:
I20260806 02:48:54.892827 25538 leader_election.cc:288] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 02:48:54.892948 25543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [term 2 FOLLOWER]: Leader election won for term 2
I20260806 02:48:54.893184 25543 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [term 2 LEADER]: Becoming Leader. State: Replica: 3f14544cf1a246e7a63bdd29f16bec3d, State: Running, Role: LEADER
I20260806 02:48:54.893666 25543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f14544cf1a246e7a63bdd29f16bec3d" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34847 } }
I20260806 02:48:54.894191 25538 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:48:54.895186 25544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3f14544cf1a246e7a63bdd29f16bec3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f14544cf1a246e7a63bdd29f16bec3d" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34847 } } }
I20260806 02:48:54.895328 25544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [sys.catalog]: This master's current role is: LEADER
I20260806 02:48:54.895803 25545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f14544cf1a246e7a63bdd29f16bec3d. Latest consensus state: current_term: 2 leader_uuid: "3f14544cf1a246e7a63bdd29f16bec3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f14544cf1a246e7a63bdd29f16bec3d" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34847 } } }
I20260806 02:48:54.895951 25545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [sys.catalog]: This master's current role is: LEADER
I20260806 02:48:54.896355 25556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:48:54.901298 25556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:48:54.901949 25556 catalog_manager.cc:1259] Loaded cluster ID: 066d508fdb7b43a180b5e3ae9155168f
I20260806 02:48:54.902014 25556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:48:54.904630 25556 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:48:54.905404 25556 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d: Loaded TSK: 0
I20260806 02:48:54.921381 25556 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d: Generated new TSK 1
I20260806 02:48:54.921675 25556 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:48:54.980930 25540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:54.981237 25540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:54.981316 25540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:54.985525 25540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:54.985612 25540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:54.985702 25540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.129
I20260806 02:48:54.991039 25540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:45809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.129:46301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.183.129
--webserver_port=39921
--tserver_master_addrs=127.1.183.190:34847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.129
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:48:54.992112 25540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:54.993193 25540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:55.004305 25576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:55.004492 25580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:55.004426 25577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:55.005365 25540 server_base.cc:1034] running on GCE node
I20260806 02:48:55.005831 25540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:55.006664 25540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:55.007874 25540 hybrid_clock.cc:648] HybridClock initialized: now 1785984535007836 us; error 46 us; skew 500 ppm
Aug 06 02:48:55 dist-test-slave-gmjp krb5kdc[24642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984535, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:55.011478 25540 init.cc:377] Logged in from keytab as kudu/127.1.183.129@KRBTEST.COM (short username kudu)
I20260806 02:48:55.012702 25540 webserver.cc:460] Webserver started at http://127.1.183.129:39921/ using document root <none> and password file <none>
I20260806 02:48:55.013453 25540 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:55.013572 25540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:55.018862 25540 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 02:48:55.021759 25589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:55.023412 25540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:48:55.023538 25540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/wal
uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b"
format_stamp: "Formatted at 2026-08-06 02:48:33 on dist-test-slave-gmjp"
server_key: "a6c81690ae02b271667092f1e57e6cd0"
server_key_iv: "fac9fe9d154cb5733cde54b2b84eca91"
server_key_version: "encryptionkey@0"
I20260806 02:48:55.024286 25540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:55.042599 25540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:55.046751 25540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:55.046969 25540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:55.047847 25540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:48:55.049520 25605 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 02:48:55.053885 25540 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 02:48:55.053967 25540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260806 02:48:55.054036 25540 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 02:48:55.056444 25540 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 02:48:55.056509 25540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260806 02:48:55.056836 25605 tablet_bootstrap.cc:492] T 45692a76b79a40049fe460bc2fbb4736 P d6dcb23d9e4f4ab49608bacef00f2a4b: Bootstrap starting.
I20260806 02:48:55.067495 25540 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.129:46301
I20260806 02:48:55.067571 25755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.129:46301 every 8 connection(s)
I20260806 02:48:55.068683 25540 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
I20260806 02:48:55.076689 25605 log.cc:824] T 45692a76b79a40049fe460bc2fbb4736 P d6dcb23d9e4f4ab49608bacef00f2a4b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:55.076867 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 25540
I20260806 02:48:55.077363 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.130:45295
--local_ip_for_outbound_sockets=127.1.183.130
--tserver_master_addrs=127.1.183.190:34847
--webserver_port=46265
--webserver_interface=127.1.183.130
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:45809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:48:55 dist-test-slave-gmjp krb5kdc[24642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984535, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:48:55.085693 25605 tablet_bootstrap.cc:492] T 45692a76b79a40049fe460bc2fbb4736 P d6dcb23d9e4f4ab49608bacef00f2a4b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:48:55.086233 25605 tablet_bootstrap.cc:492] T 45692a76b79a40049fe460bc2fbb4736 P d6dcb23d9e4f4ab49608bacef00f2a4b: Bootstrap complete.
I20260806 02:48:55.087333 25759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:55.070944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:55113 (local address 127.1.183.190:34847)
0806 02:48:55.071330 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:55.071353 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:55.071681 (+ 328us) server_negotiation.cc:407] Connection header received
0806 02:48:55.072878 (+ 1197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:55.072888 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:55.073172 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:55.073512 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:55.074802 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.075762 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.077039 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.077866 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.078084 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:55.081109 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:55.081115 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:55.081132 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:55.081167 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:55.083417 (+ 2250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:55.084133 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:55.084139 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:55.084145 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:55.084322 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:55.084777 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:55.084780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:55.084782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:55.085230 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:55.085377 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:55.086100 (+ 723us) server_negotiation.cc:299] Negotiation successful
0806 02:48:55.086431 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":133,"threads_started":1}
I20260806 02:48:55.087404 25760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:55.070953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.129:55113)
0806 02:48:55.071508 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:55.071562 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:55.072459 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:55.073572 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:55.073580 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:55.074024 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:55.074479 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:55.074493 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.075921 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.075929 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:55.076894 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:55.076907 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.078031 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.078035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:55.078223 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:55.078886 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:55.078903 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:55.080868 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:55.083556 (+ 2688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:55.083565 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:55.083582 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:55.083980 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:55.084443 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:55.084449 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:55.084454 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:55.084648 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:55.085366 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:55.085375 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:55.085708 (+ 333us) client_negotiation.cc:769] Sending connection context
0806 02:48:55.085938 (+ 230us) client_negotiation.cc:240] Negotiation successful
0806 02:48:55.086319 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":131,"threads_started":1}
I20260806 02:48:55.088093 25605 ts_tablet_manager.cc:1397] T 45692a76b79a40049fe460bc2fbb4736 P d6dcb23d9e4f4ab49608bacef00f2a4b: Time spent bootstrapping tablet: real 0.032s user 0.021s sys 0.009s
I20260806 02:48:55.090142 25756 heartbeater.cc:347] Connected to a master server at 127.1.183.190:34847
I20260806 02:48:55.090406 25756 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:55.091014 25756 heartbeater.cc:511] Master 127.1.183.190:34847 requested a full tablet report, sending...
I20260806 02:48:55.090902 25605 raft_consensus.cc:348] T 45692a76b79a40049fe460bc2fbb4736 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984378e530c349159d547c20dfef131f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 45295 } } peers { permanent_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 42193 } } peers { permanent_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46301 } }
I20260806 02:48:55.091389 25605 raft_consensus.cc:729] T 45692a76b79a40049fe460bc2fbb4736 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6dcb23d9e4f4ab49608bacef00f2a4b, State: Initialized, Role: FOLLOWER
I20260806 02:48:55.091883 25605 consensus_queue.cc:260] T 45692a76b79a40049fe460bc2fbb4736 P d6dcb23d9e4f4ab49608bacef00f2a4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984378e530c349159d547c20dfef131f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 45295 } } peers { permanent_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 42193 } } peers { permanent_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46301 } }
I20260806 02:48:55.092921 25605 ts_tablet_manager.cc:1428] T 45692a76b79a40049fe460bc2fbb4736 P d6dcb23d9e4f4ab49608bacef00f2a4b: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260806 02:48:55.094623 25486 ts_manager.cc:194] Registered new tserver with Master: d6dcb23d9e4f4ab49608bacef00f2a4b (127.1.183.129:46301)
I20260806 02:48:55.098380 25486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.129@KRBTEST.COM'} at 127.1.183.129:55113
I20260806 02:48:55.101143 25756 heartbeater.cc:503] Master 127.1.183.190:34847 was elected leader, sending a full tablet report...
I20260806 02:48:55.105268 25770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:55.097696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:46301 (local address 127.0.0.1:58282)
0806 02:48:55.098596 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:55.098644 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:55.098781 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:55.100953 (+ 2172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:55.100959 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:55.100986 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:55.101137 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:55.101146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.102497 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.102500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:55.103729 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:55.103740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.104808 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.104811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:55.104907 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:55.104939 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 02:48:55.105035 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 02:48:55.105135 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":149,"threads_started":1}
I20260806 02:48:55.105321 25771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:55.097807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58282 (local address 127.1.183.129:46301)
0806 02:48:55.100413 (+ 2606us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:55.100427 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:55.100453 (+ 26us) server_negotiation.cc:407] Connection header received
0806 02:48:55.100532 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:55.100537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:55.100752 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:55.100927 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:55.101275 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.102366 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.103864 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.104656 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.104774 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:55.104927 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:55.105084 (+ 157us) server_negotiation.cc:299] Negotiation successful
0806 02:48:55.105180 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":2485,"thread_start_us":111,"threads_started":1}
I20260806 02:48:55.106499 25658 tablet_service.cc:1514] Processing DeleteTablet for tablet 45692a76b79a40049fe460bc2fbb4736 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:48:34 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:58282
I20260806 02:48:55.107542 25772 tablet_replica.cc:330] stopping tablet replica
I20260806 02:48:55.107882 25772 raft_consensus.cc:2229] T 45692a76b79a40049fe460bc2fbb4736 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:48:55.108203 25772 raft_consensus.cc:2258] T 45692a76b79a40049fe460bc2fbb4736 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:48:55.109354 25772 ts_tablet_manager.cc:1905] T 45692a76b79a40049fe460bc2fbb4736 P d6dcb23d9e4f4ab49608bacef00f2a4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:48:55.113020 25772 ts_tablet_manager.cc:1918] T 45692a76b79a40049fe460bc2fbb4736 P d6dcb23d9e4f4ab49608bacef00f2a4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:48:55.113149 25772 log.cc:1196] T 45692a76b79a40049fe460bc2fbb4736 P d6dcb23d9e4f4ab49608bacef00f2a4b: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/wal/wals/45692a76b79a40049fe460bc2fbb4736
I20260806 02:48:55.113485 25772 ts_tablet_manager.cc:1939] T 45692a76b79a40049fe460bc2fbb4736 P d6dcb23d9e4f4ab49608bacef00f2a4b: Deleting consensus metadata
I20260806 02:48:55.115283 25467 catalog_manager.cc:4841] TS d6dcb23d9e4f4ab49608bacef00f2a4b (127.1.183.129:46301): tablet 45692a76b79a40049fe460bc2fbb4736 (table test-table [id=305b440421e5442483245b9453762240]) successfully deleted
W20260806 02:48:55.203904 25766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:55.204246 25766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:55.204329 25766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:55.208606 25766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:55.208691 25766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:55.208783 25766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.130
I20260806 02:48:55.214264 25766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:45809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.130:45295
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.183.130
--webserver_port=46265
--tserver_master_addrs=127.1.183.190:34847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.130
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:48:55.215490 25766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:55.216681 25766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:55.226716 25782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:55.226681 25778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:55.226681 25779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:55.226861 25766 server_base.cc:1034] running on GCE node
I20260806 02:48:55.227496 25766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:55.228207 25766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:55.229442 25766 hybrid_clock.cc:648] HybridClock initialized: now 1785984535229424 us; error 30 us; skew 500 ppm
Aug 06 02:48:55 dist-test-slave-gmjp krb5kdc[24642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984535, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:55.232451 25766 init.cc:377] Logged in from keytab as kudu/127.1.183.130@KRBTEST.COM (short username kudu)
I20260806 02:48:55.233330 25766 webserver.cc:460] Webserver started at http://127.1.183.130:46265/ using document root <none> and password file <none>
I20260806 02:48:55.233917 25766 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:55.234035 25766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:55.238049 25766 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:48:55.240296 25794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:55.241531 25766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 02:48:55.241672 25766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/wal
uuid: "984378e530c349159d547c20dfef131f"
format_stamp: "Formatted at 2026-08-06 02:48:33 on dist-test-slave-gmjp"
server_key: "80db4ce4d4cd2f3544e1f74b38c73a38"
server_key_iv: "c0558eb2182cb3d003d28da14c166b42"
server_key_version: "encryptionkey@0"
I20260806 02:48:55.242235 25766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:55.255427 25766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:55.258965 25766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:55.259186 25766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:55.259848 25766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:48:55.261073 25802 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260806 02:48:55.265501 25802 ts_tablet_manager.cc:1852] T 45692a76b79a40049fe460bc2fbb4736 P 984378e530c349159d547c20dfef131f: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260806 02:48:55.265992 25802 ts_tablet_manager.cc:1905] T 45692a76b79a40049fe460bc2fbb4736 P 984378e530c349159d547c20dfef131f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:48:55.270591 25802 ts_tablet_manager.cc:1918] T 45692a76b79a40049fe460bc2fbb4736 P 984378e530c349159d547c20dfef131f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:48:55.270735 25802 log.cc:1196] T 45692a76b79a40049fe460bc2fbb4736 P 984378e530c349159d547c20dfef131f: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/wal/wals/45692a76b79a40049fe460bc2fbb4736
I20260806 02:48:55.271040 25802 ts_tablet_manager.cc:1939] T 45692a76b79a40049fe460bc2fbb4736 P 984378e530c349159d547c20dfef131f: Deleting consensus metadata
I20260806 02:48:55.271436 25766 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260806 02:48:55.271520 25766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s user 0.000s sys 0.000s
I20260806 02:48:55.271636 25766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:48:55.271680 25766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:55.281656 25766 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.130:45295
I20260806 02:48:55.281693 25965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.130:45295 every 8 connection(s)
I20260806 02:48:55.282624 25766 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
I20260806 02:48:55.286564 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 25766
I20260806 02:48:55.287036 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.131:42193
--local_ip_for_outbound_sockets=127.1.183.131
--tserver_master_addrs=127.1.183.190:34847
--webserver_port=33283
--webserver_interface=127.1.183.131
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:45809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:48:55 dist-test-slave-gmjp krb5kdc[24642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984535, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:48:55.297683 25759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:55.284707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:49839 (local address 127.1.183.190:34847)
0806 02:48:55.284898 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:55.284902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:55.285848 (+ 946us) server_negotiation.cc:407] Connection header received
0806 02:48:55.286914 (+ 1066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:55.286919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:55.286980 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:55.287060 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:55.288346 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.288889 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.290088 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.290353 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.290470 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:55.293492 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:55.293496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:55.293498 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:55.293527 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:55.295196 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:55.295868 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:55.295872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:55.295874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:55.295966 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:55.296428 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:55.296432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:55.296434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:55.296639 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:55.296751 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:55.297342 (+ 591us) server_negotiation.cc:299] Negotiation successful
0806 02:48:55.297496 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 02:48:55.298578 25969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:55.285132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.130:49839)
0806 02:48:55.285637 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:55.285682 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:55.286670 (+ 988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:55.287241 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:55.287249 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:55.287615 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:55.288110 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:55.288124 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.289031 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.289034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:55.289948 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:55.289958 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.290532 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.290536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:55.290718 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:55.291453 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:55.291473 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:55.293264 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:55.295369 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:55.295377 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:55.295398 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:55.295692 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:55.296075 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:55.296078 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:55.296081 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:55.296325 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:55.296752 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:55.296761 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:55.297067 (+ 306us) client_negotiation.cc:769] Sending connection context
0806 02:48:55.297270 (+ 203us) client_negotiation.cc:240] Negotiation successful
0806 02:48:55.297647 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":122,"threads_started":1}
I20260806 02:48:55.299942 25967 heartbeater.cc:347] Connected to a master server at 127.1.183.190:34847
I20260806 02:48:55.300276 25967 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:55.300899 25967 heartbeater.cc:511] Master 127.1.183.190:34847 requested a full tablet report, sending...
I20260806 02:48:55.302202 25486 ts_manager.cc:194] Registered new tserver with Master: 984378e530c349159d547c20dfef131f (127.1.183.130:45295)
I20260806 02:48:55.302835 25486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.130@KRBTEST.COM'} at 127.1.183.130:49839
W20260806 02:48:55.409825 25971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:48:55.410127 25971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:48:55.410173 25971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:48:55.414376 25971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:48:55.414469 25971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:48:55.414587 25971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.131
I20260806 02:48:55.420001 25971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:45809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.131:42193
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.183.131
--webserver_port=33283
--tserver_master_addrs=127.1.183.190:34847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.131
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:48:55.421173 25971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:48:55.422253 25971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:48:55.432737 25981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:48:55.432713 25977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:55.432891 25971 server_base.cc:1034] running on GCE node
W20260806 02:48:55.432703 25978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:48:55.433388 25971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:48:55.434057 25971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:48:55.435266 25971 hybrid_clock.cc:648] HybridClock initialized: now 1785984535435249 us; error 25 us; skew 500 ppm
Aug 06 02:48:55 dist-test-slave-gmjp krb5kdc[24642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984535, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:48:55.438341 25971 init.cc:377] Logged in from keytab as kudu/127.1.183.131@KRBTEST.COM (short username kudu)
I20260806 02:48:55.439252 25971 webserver.cc:460] Webserver started at http://127.1.183.131:33283/ using document root <none> and password file <none>
I20260806 02:48:55.439834 25971 fs_manager.cc:359] Metadata directory not provided
I20260806 02:48:55.439953 25971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:48:55.443820 25971 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:48:55.446156 25993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:55.447549 25971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 02:48:55.447691 25971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/wal
uuid: "aad0f35208db44fe8cf5fcc4b6a00a89"
format_stamp: "Formatted at 2026-08-06 02:48:34 on dist-test-slave-gmjp"
server_key: "c9be08837258de05c57d727320032012"
server_key_iv: "a6ce26e6205e3eec22eec21ad4c27bbe"
server_key_version: "encryptionkey@0"
I20260806 02:48:55.448292 25971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:48:55.468135 25971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:48:55.471429 25971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:48:55.471652 25971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:48:55.472343 25971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:48:55.473227 25971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:48:55.473275 25971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:55.473315 25971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:48:55.473369 25971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:48:55.482697 25971 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.131:42193
I20260806 02:48:55.482784 26151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.131:42193 every 8 connection(s)
I20260806 02:48:55.483765 25971 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
I20260806 02:48:55.485773 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 25971
Aug 06 02:48:55 dist-test-slave-gmjp krb5kdc[24642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984535, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:48:55.494778 26156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:55.486479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46002 (local address 127.1.183.190:34847)
0806 02:48:55.486813 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:55.486816 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:55.486905 (+ 89us) server_negotiation.cc:407] Connection header received
0806 02:48:55.487050 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:55.487052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:55.487095 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:55.487203 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:55.488005 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.488610 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.489493 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.489719 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.489835 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:55.491243 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:55.491247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:55.491249 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:55.491280 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:55.492953 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:55.493398 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:55.493403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:55.493405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:55.493489 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:55.493774 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:55.493778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:55.493779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:55.493995 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:55.494216 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:55.494440 (+ 224us) server_negotiation.cc:299] Negotiation successful
0806 02:48:55.494564 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":106,"threads_started":1}
I20260806 02:48:55.498430 25759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:55.485806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:41725 (local address 127.1.183.190:34847)
0806 02:48:55.485985 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:55.485989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:55.487670 (+ 1681us) server_negotiation.cc:407] Connection header received
0806 02:48:55.488658 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:55.488661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:55.488710 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:55.488818 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:55.489978 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.490470 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.491617 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.491831 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.491943 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:55.494479 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:55.494482 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:55.494484 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:55.494508 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:55.496218 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:55.496764 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:55.496768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:55.496802 (+ 34us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:55.496914 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:55.497318 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:55.497320 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:55.497321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:55.497481 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:55.497560 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:55.498212 (+ 652us) server_negotiation.cc:299] Negotiation successful
0806 02:48:55.498313 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260806 02:48:55.499248 26157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:55.486364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.131:41725)
0806 02:48:55.487526 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:55.487567 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:55.488459 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:55.488997 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:55.489004 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:55.489363 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:48:55.489839 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:55.489852 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.490598 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.490607 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:55.491476 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:55.491485 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.491982 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.491986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:55.492225 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:55.492736 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:48:55.492752 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:48:55.494307 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:48:55.496370 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:55.496376 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:55.496392 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:55.496655 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:55.497019 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:48:55.497021 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:48:55.497023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:48:55.497223 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:48:55.497579 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:48:55.497584 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:48:55.497866 (+ 282us) client_negotiation.cc:769] Sending connection context
0806 02:48:55.498030 (+ 164us) client_negotiation.cc:240] Negotiation successful
0806 02:48:55.498320 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"thread_start_us":150,"threads_started":1}
I20260806 02:48:55.500418 26152 heartbeater.cc:347] Connected to a master server at 127.1.183.190:34847
I20260806 02:48:55.500674 26152 heartbeater.cc:464] Registering TS with master...
I20260806 02:48:55.501180 26152 heartbeater.cc:511] Master 127.1.183.190:34847 requested a full tablet report, sending...
I20260806 02:48:55.502210 25486 ts_manager.cc:194] Registered new tserver with Master: aad0f35208db44fe8cf5fcc4b6a00a89 (127.1.183.131:42193)
I20260806 02:48:55.502780 25486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.131@KRBTEST.COM'} at 127.1.183.131:41725
I20260806 02:48:55.509207 1758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:48:55.513916 25759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:55.509910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46014 (local address 127.1.183.190:34847)
0806 02:48:55.510070 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:55.510076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:55.510244 (+ 168us) server_negotiation.cc:407] Connection header received
0806 02:48:55.510479 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:55.510482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:55.510595 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:55.510735 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:48:55.511044 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.511587 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.512620 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.512843 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.512963 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:55.513311 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:48:55.513458 (+ 147us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0806 02:48:55.513751 (+ 293us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46014: token expired
Metrics: {"server-negotiator.queue_time_us":59}
W20260806 02:48:55.514015 25759 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46014: token expired
W20260806 02:48:55.514284 25386 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.183.190:34847)
I20260806 02:48:55.522341 25759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:55.514678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46024 (local address 127.1.183.190:34847)
0806 02:48:55.514800 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:55.514807 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:55.514874 (+ 67us) server_negotiation.cc:407] Connection header received
0806 02:48:55.515018 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:55.515022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:55.515072 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:55.515162 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:48:55.515840 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.516398 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.517223 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.517434 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.517555 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:55.519010 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:48:55.519015 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:48:55.519018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:48:55.519052 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:48:55.520485 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:55.520954 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:55.520959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:55.520961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:55.521053 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:48:55.521363 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:48:55.521368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:48:55.521370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:48:55.521599 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:48:55.521761 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:55.522025 (+ 264us) server_negotiation.cc:299] Negotiation successful
0806 02:48:55.522146 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:48:55.525025 25486 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:48:55.528347 25486 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:48:55.536844 25658 tablet_service.cc:1467] Processing CreateTablet for tablet 1375c5d029404611a4e7ddb3d9e1cea1 (DEFAULT_TABLE table=test-table [id=1ea4db1ded8f4b05839f2b9d8a74c476]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:48:55.537598 25658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1375c5d029404611a4e7ddb3d9e1cea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:48:55.540668 25605 tablet_bootstrap.cc:492] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b: Bootstrap starting.
I20260806 02:48:55.541643 25605 tablet_bootstrap.cc:654] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:55.541788 25770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:55.535425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:45295 (local address 127.0.0.1:45924)
0806 02:48:55.535555 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:55.535572 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:55.535655 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:55.537038 (+ 1383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:55.537042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:55.537063 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:55.537160 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:55.537168 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.538897 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.538903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:55.540077 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:55.540092 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.541330 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.541334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:55.541422 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:55.541449 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 02:48:55.541539 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 02:48:55.541644 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":3}
I20260806 02:48:55.541853 26162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:55.535652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45924 (local address 127.1.183.130:45295)
0806 02:48:55.536622 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:55.536633 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:55.536653 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:48:55.536718 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:55.536722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:55.536880 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:55.537328 (+ 448us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:55.537525 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.538767 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.540245 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.541196 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.541308 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:55.541475 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:55.541571 (+ 96us) server_negotiation.cc:299] Negotiation successful
0806 02:48:55.541672 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":874,"thread_start_us":147,"threads_started":1}
I20260806 02:48:55.543033 26166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:55.536044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54968 (local address 127.1.183.131:42193)
0806 02:48:55.536431 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:55.536446 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:55.536476 (+ 30us) server_negotiation.cc:407] Connection header received
0806 02:48:55.536564 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:55.536570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:55.536804 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:55.536989 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:55.538403 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.539461 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.540901 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.542434 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.542552 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:55.542692 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:55.542784 (+ 92us) server_negotiation.cc:299] Negotiation successful
0806 02:48:55.542893 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":164,"threads_started":1}
I20260806 02:48:55.543177 25605 tablet_bootstrap.cc:492] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b: No bootstrap required, opened a new log
I20260806 02:48:55.543262 25605 ts_tablet_manager.cc:1397] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260806 02:48:55.543589 26165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:55.535883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:42193 (local address 127.0.0.1:54968)
0806 02:48:55.536111 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:55.536127 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:55.536259 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:55.536973 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:55.536976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:55.536996 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:55.537111 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:55.537117 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.539592 (+ 2475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.539597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:55.540744 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:55.540756 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.542565 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.542569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:55.542659 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:55.542680 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 02:48:55.542754 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 02:48:55.543448 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":94,"threads_started":1}
I20260806 02:48:55.543614 25605 raft_consensus.cc:348] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984378e530c349159d547c20dfef131f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 45295 } } peers { permanent_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 42193 } } peers { permanent_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46301 } }
I20260806 02:48:55.543781 25605 raft_consensus.cc:374] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:55.543821 25605 raft_consensus.cc:729] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6dcb23d9e4f4ab49608bacef00f2a4b, State: Initialized, Role: FOLLOWER
I20260806 02:48:55.543514 25862 tablet_service.cc:1467] Processing CreateTablet for tablet 1375c5d029404611a4e7ddb3d9e1cea1 (DEFAULT_TABLE table=test-table [id=1ea4db1ded8f4b05839f2b9d8a74c476]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:48:55.543991 25605 consensus_queue.cc:260] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984378e530c349159d547c20dfef131f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 45295 } } peers { permanent_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 42193 } } peers { permanent_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46301 } }
I20260806 02:48:55.544299 25605 ts_tablet_manager.cc:1428] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 02:48:55.544697 25862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1375c5d029404611a4e7ddb3d9e1cea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:48:55.546723 26057 tablet_service.cc:1467] Processing CreateTablet for tablet 1375c5d029404611a4e7ddb3d9e1cea1 (DEFAULT_TABLE table=test-table [id=1ea4db1ded8f4b05839f2b9d8a74c476]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:48:55.547801 26057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1375c5d029404611a4e7ddb3d9e1cea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:48:55.550141 25802 tablet_bootstrap.cc:492] T 1375c5d029404611a4e7ddb3d9e1cea1 P 984378e530c349159d547c20dfef131f: Bootstrap starting.
I20260806 02:48:55.552430 25802 tablet_bootstrap.cc:654] T 1375c5d029404611a4e7ddb3d9e1cea1 P 984378e530c349159d547c20dfef131f: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:55.553388 25802 log.cc:824] T 1375c5d029404611a4e7ddb3d9e1cea1 P 984378e530c349159d547c20dfef131f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:55.555234 25802 tablet_bootstrap.cc:492] T 1375c5d029404611a4e7ddb3d9e1cea1 P 984378e530c349159d547c20dfef131f: No bootstrap required, opened a new log
I20260806 02:48:55.555387 26171 tablet_bootstrap.cc:492] T 1375c5d029404611a4e7ddb3d9e1cea1 P aad0f35208db44fe8cf5fcc4b6a00a89: Bootstrap starting.
I20260806 02:48:55.555493 25802 ts_tablet_manager.cc:1397] T 1375c5d029404611a4e7ddb3d9e1cea1 P 984378e530c349159d547c20dfef131f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:48:55.558003 26171 tablet_bootstrap.cc:654] T 1375c5d029404611a4e7ddb3d9e1cea1 P aad0f35208db44fe8cf5fcc4b6a00a89: Neither blocks nor log segments found. Creating new log.
I20260806 02:48:55.558796 26171 log.cc:824] T 1375c5d029404611a4e7ddb3d9e1cea1 P aad0f35208db44fe8cf5fcc4b6a00a89: Log is configured to *not* fsync() on all Append() calls
I20260806 02:48:55.559105 25802 raft_consensus.cc:348] T 1375c5d029404611a4e7ddb3d9e1cea1 P 984378e530c349159d547c20dfef131f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984378e530c349159d547c20dfef131f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 45295 } } peers { permanent_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 42193 } } peers { permanent_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46301 } }
I20260806 02:48:55.559357 25802 raft_consensus.cc:374] T 1375c5d029404611a4e7ddb3d9e1cea1 P 984378e530c349159d547c20dfef131f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:55.559437 25802 raft_consensus.cc:729] T 1375c5d029404611a4e7ddb3d9e1cea1 P 984378e530c349159d547c20dfef131f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 984378e530c349159d547c20dfef131f, State: Initialized, Role: FOLLOWER
I20260806 02:48:55.559895 25802 consensus_queue.cc:260] T 1375c5d029404611a4e7ddb3d9e1cea1 P 984378e530c349159d547c20dfef131f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984378e530c349159d547c20dfef131f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 45295 } } peers { permanent_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 42193 } } peers { permanent_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46301 } }
I20260806 02:48:55.560631 25967 heartbeater.cc:503] Master 127.1.183.190:34847 was elected leader, sending a full tablet report...
I20260806 02:48:55.560909 26171 tablet_bootstrap.cc:492] T 1375c5d029404611a4e7ddb3d9e1cea1 P aad0f35208db44fe8cf5fcc4b6a00a89: No bootstrap required, opened a new log
I20260806 02:48:55.560946 25802 ts_tablet_manager.cc:1428] T 1375c5d029404611a4e7ddb3d9e1cea1 P 984378e530c349159d547c20dfef131f: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260806 02:48:55.561060 26171 ts_tablet_manager.cc:1397] T 1375c5d029404611a4e7ddb3d9e1cea1 P aad0f35208db44fe8cf5fcc4b6a00a89: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260806 02:48:55.564664 26171 raft_consensus.cc:348] T 1375c5d029404611a4e7ddb3d9e1cea1 P aad0f35208db44fe8cf5fcc4b6a00a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984378e530c349159d547c20dfef131f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 45295 } } peers { permanent_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 42193 } } peers { permanent_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46301 } }
I20260806 02:48:55.564913 26171 raft_consensus.cc:374] T 1375c5d029404611a4e7ddb3d9e1cea1 P aad0f35208db44fe8cf5fcc4b6a00a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:48:55.564960 26171 raft_consensus.cc:729] T 1375c5d029404611a4e7ddb3d9e1cea1 P aad0f35208db44fe8cf5fcc4b6a00a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad0f35208db44fe8cf5fcc4b6a00a89, State: Initialized, Role: FOLLOWER
I20260806 02:48:55.565310 26171 consensus_queue.cc:260] T 1375c5d029404611a4e7ddb3d9e1cea1 P aad0f35208db44fe8cf5fcc4b6a00a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984378e530c349159d547c20dfef131f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 45295 } } peers { permanent_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 42193 } } peers { permanent_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46301 } }
I20260806 02:48:55.565948 26152 heartbeater.cc:503] Master 127.1.183.190:34847 was elected leader, sending a full tablet report...
I20260806 02:48:55.566250 26171 ts_tablet_manager.cc:1428] T 1375c5d029404611a4e7ddb3d9e1cea1 P aad0f35208db44fe8cf5fcc4b6a00a89: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260806 02:48:55.570298 25758 tablet.cc:2369] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:48:55.600299 26175 raft_consensus.cc:482] T 1375c5d029404611a4e7ddb3d9e1cea1 P aad0f35208db44fe8cf5fcc4b6a00a89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:48:55.600507 26175 raft_consensus.cc:504] T 1375c5d029404611a4e7ddb3d9e1cea1 P aad0f35208db44fe8cf5fcc4b6a00a89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984378e530c349159d547c20dfef131f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 45295 } } peers { permanent_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 42193 } } peers { permanent_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46301 } }
I20260806 02:48:55.601676 26175 leader_election.cc:288] T 1375c5d029404611a4e7ddb3d9e1cea1 P aad0f35208db44fe8cf5fcc4b6a00a89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 984378e530c349159d547c20dfef131f (127.1.183.130:45295), d6dcb23d9e4f4ab49608bacef00f2a4b (127.1.183.129:46301)
I20260806 02:48:55.606869 25771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:55.602404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:55701 (local address 127.1.183.129:46301)
0806 02:48:55.602532 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:55.602536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:55.602551 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:48:55.602718 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:55.602722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:55.602773 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:55.602853 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:55.603151 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.603862 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.605557 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.606168 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.606305 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:55.606367 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:55.606664 (+ 297us) server_negotiation.cc:299] Negotiation successful
0806 02:48:55.606743 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:48:55.606869 26162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:55.602054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:35939 (local address 127.1.183.130:45295)
0806 02:48:55.602174 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:55.602178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:55.602192 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:48:55.602335 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:55.602338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:55.602392 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:55.602476 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:48:55.602931 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.603717 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.605646 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.606252 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.606357 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:55.606417 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:55.606661 (+ 244us) server_negotiation.cc:299] Negotiation successful
0806 02:48:55.606736 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:48:55.606899 26157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:55.601875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:45295 (local address 127.1.183.131:35939)
0806 02:48:55.602079 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:55.602096 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:55.602197 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:55.602606 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:55.602609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:55.602635 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:55.602784 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:55.602792 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.603864 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.603867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:55.605413 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:55.605425 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.606351 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.606354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:55.606453 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:55.606559 (+ 106us) client_negotiation.cc:769] Sending connection context
0806 02:48:55.606631 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 02:48:55.606726 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":169}
I20260806 02:48:55.606912 26176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:55.602022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:46301 (local address 127.1.183.131:55701)
0806 02:48:55.602476 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:55.602491 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:48:55.602594 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:48:55.602879 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:48:55.602885 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:48:55.602907 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:48:55.603036 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:55.603041 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.603966 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.603969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:55.605413 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:48:55.605425 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.606321 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.606326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:48:55.606435 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:55.606532 (+ 97us) client_negotiation.cc:769] Sending connection context
0806 02:48:55.606621 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 02:48:55.606726 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":143,"thread_start_us":131,"threads_started":1}
I20260806 02:48:55.607595 25693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1375c5d029404611a4e7ddb3d9e1cea1" candidate_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b" is_pre_election: true
I20260806 02:48:55.607784 25892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1375c5d029404611a4e7ddb3d9e1cea1" candidate_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "984378e530c349159d547c20dfef131f" is_pre_election: true
I20260806 02:48:55.608008 25693 raft_consensus.cc:2454] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad0f35208db44fe8cf5fcc4b6a00a89 in term 0.
I20260806 02:48:55.608058 25892 raft_consensus.cc:2454] T 1375c5d029404611a4e7ddb3d9e1cea1 P 984378e530c349159d547c20dfef131f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad0f35208db44fe8cf5fcc4b6a00a89 in term 0.
I20260806 02:48:55.608598 25994 leader_election.cc:302] T 1375c5d029404611a4e7ddb3d9e1cea1 P aad0f35208db44fe8cf5fcc4b6a00a89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 984378e530c349159d547c20dfef131f, aad0f35208db44fe8cf5fcc4b6a00a89, d6dcb23d9e4f4ab49608bacef00f2a4b; no voters:
I20260806 02:48:55.608933 26175 raft_consensus.cc:2781] T 1375c5d029404611a4e7ddb3d9e1cea1 P aad0f35208db44fe8cf5fcc4b6a00a89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:48:55.609054 26175 raft_consensus.cc:482] T 1375c5d029404611a4e7ddb3d9e1cea1 P aad0f35208db44fe8cf5fcc4b6a00a89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:48:55.609112 26175 raft_consensus.cc:3037] T 1375c5d029404611a4e7ddb3d9e1cea1 P aad0f35208db44fe8cf5fcc4b6a00a89 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:55.610285 26175 raft_consensus.cc:504] T 1375c5d029404611a4e7ddb3d9e1cea1 P aad0f35208db44fe8cf5fcc4b6a00a89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984378e530c349159d547c20dfef131f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 45295 } } peers { permanent_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 42193 } } peers { permanent_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46301 } }
I20260806 02:48:55.610641 26175 leader_election.cc:288] T 1375c5d029404611a4e7ddb3d9e1cea1 P aad0f35208db44fe8cf5fcc4b6a00a89 [CANDIDATE]: Term 1 election: Requested vote from peers 984378e530c349159d547c20dfef131f (127.1.183.130:45295), d6dcb23d9e4f4ab49608bacef00f2a4b (127.1.183.129:46301)
I20260806 02:48:55.611126 25892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1375c5d029404611a4e7ddb3d9e1cea1" candidate_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "984378e530c349159d547c20dfef131f"
I20260806 02:48:55.611126 25693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1375c5d029404611a4e7ddb3d9e1cea1" candidate_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b"
I20260806 02:48:55.611265 25693 raft_consensus.cc:3037] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:55.611265 25892 raft_consensus.cc:3037] T 1375c5d029404611a4e7ddb3d9e1cea1 P 984378e530c349159d547c20dfef131f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:48:55.612654 25892 raft_consensus.cc:2454] T 1375c5d029404611a4e7ddb3d9e1cea1 P 984378e530c349159d547c20dfef131f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad0f35208db44fe8cf5fcc4b6a00a89 in term 1.
I20260806 02:48:55.612654 25693 raft_consensus.cc:2454] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad0f35208db44fe8cf5fcc4b6a00a89 in term 1.
I20260806 02:48:55.612965 25994 leader_election.cc:302] T 1375c5d029404611a4e7ddb3d9e1cea1 P aad0f35208db44fe8cf5fcc4b6a00a89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 984378e530c349159d547c20dfef131f, aad0f35208db44fe8cf5fcc4b6a00a89, d6dcb23d9e4f4ab49608bacef00f2a4b; no voters:
I20260806 02:48:55.613118 26175 raft_consensus.cc:2781] T 1375c5d029404611a4e7ddb3d9e1cea1 P aad0f35208db44fe8cf5fcc4b6a00a89 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:48:55.613348 26175 raft_consensus.cc:686] T 1375c5d029404611a4e7ddb3d9e1cea1 P aad0f35208db44fe8cf5fcc4b6a00a89 [term 1 LEADER]: Becoming Leader. State: Replica: aad0f35208db44fe8cf5fcc4b6a00a89, State: Running, Role: LEADER
I20260806 02:48:55.613734 26175 consensus_queue.cc:237] T 1375c5d029404611a4e7ddb3d9e1cea1 P aad0f35208db44fe8cf5fcc4b6a00a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984378e530c349159d547c20dfef131f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 45295 } } peers { permanent_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 42193 } } peers { permanent_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46301 } }
I20260806 02:48:55.616899 25485 catalog_manager.cc:5495] T 1375c5d029404611a4e7ddb3d9e1cea1 P aad0f35208db44fe8cf5fcc4b6a00a89 reported cstate change: term changed from 0 to 1, leader changed from <none> to aad0f35208db44fe8cf5fcc4b6a00a89 (127.1.183.131). New cstate: current_term: 1 leader_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984378e530c349159d547c20dfef131f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 45295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 42193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46301 } health_report { overall_health: UNKNOWN } } }
I20260806 02:48:55.646457 26166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:55.642765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54972 (local address 127.1.183.131:42193)
0806 02:48:55.642902 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 02:48:55.642917 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:48:55.642931 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:48:55.643037 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:48:55.643040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:48:55.643094 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:48:55.643187 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:48:55.643470 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.643978 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.644868 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:48:55.645096 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:48:55.645173 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:48:55.645537 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:48:55.645985 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:48:55.646131 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 02:48:55.646200 (+ 69us) server_negotiation.cc:299] Negotiation successful
0806 02:48:55.646266 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:48:55.652736 25693 raft_consensus.cc:1261] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 1 FOLLOWER]: Refusing update from remote peer aad0f35208db44fe8cf5fcc4b6a00a89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:48:55.652736 25892 raft_consensus.cc:1261] T 1375c5d029404611a4e7ddb3d9e1cea1 P 984378e530c349159d547c20dfef131f [term 1 FOLLOWER]: Refusing update from remote peer aad0f35208db44fe8cf5fcc4b6a00a89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:48:55.653458 26175 consensus_queue.cc:1035] T 1375c5d029404611a4e7ddb3d9e1cea1 P aad0f35208db44fe8cf5fcc4b6a00a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:48:55.653707 26178 consensus_queue.cc:1035] T 1375c5d029404611a4e7ddb3d9e1cea1 P aad0f35208db44fe8cf5fcc4b6a00a89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "984378e530c349159d547c20dfef131f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 45295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:48:55.659142 26180 mvcc.cc:205] Tried to move back new op lower bound from 7315392658025525248 to 7315392657876398080. Current Snapshot: MvccSnapshot[applied={T|T < 7315392658025525248}]
I20260806 02:48:55.661453 26181 mvcc.cc:205] Tried to move back new op lower bound from 7315392658025525248 to 7315392657876398080. Current Snapshot: MvccSnapshot[applied={T|T < 7315392658025525248}]
I20260806 02:48:55.669454 25486 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:48:55.669648 25486 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:48:55.672088 25486 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d: Sending DeleteTablet for 3 replicas of tablet 1375c5d029404611a4e7ddb3d9e1cea1
I20260806 02:48:55.672909 26057 tablet_service.cc:1514] Processing DeleteTablet for tablet 1375c5d029404611a4e7ddb3d9e1cea1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:48:55 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:54968
I20260806 02:48:55.672977 25658 tablet_service.cc:1514] Processing DeleteTablet for tablet 1375c5d029404611a4e7ddb3d9e1cea1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:48:55 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:58282
I20260806 02:48:55.673357 26191 tablet_replica.cc:330] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b: stopping tablet replica
I20260806 02:48:55.673415 26192 tablet_replica.cc:330] T 1375c5d029404611a4e7ddb3d9e1cea1 P aad0f35208db44fe8cf5fcc4b6a00a89: stopping tablet replica
I20260806 02:48:55.673487 26191 raft_consensus.cc:2229] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:48:55.673480 25862 tablet_service.cc:1514] Processing DeleteTablet for tablet 1375c5d029404611a4e7ddb3d9e1cea1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:48:55 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:45924
I20260806 02:48:55.673614 26191 raft_consensus.cc:2258] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:48:55.673849 26192 raft_consensus.cc:2229] T 1375c5d029404611a4e7ddb3d9e1cea1 P aad0f35208db44fe8cf5fcc4b6a00a89 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:48:55.674240 26192 raft_consensus.cc:2258] T 1375c5d029404611a4e7ddb3d9e1cea1 P aad0f35208db44fe8cf5fcc4b6a00a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:48:55.674448 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 25540
I20260806 02:48:55.674566 26191 ts_tablet_manager.cc:1905] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:48:55.683267 26192 ts_tablet_manager.cc:1905] T 1375c5d029404611a4e7ddb3d9e1cea1 P aad0f35208db44fe8cf5fcc4b6a00a89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:48:55.686658 26192 ts_tablet_manager.cc:1918] T 1375c5d029404611a4e7ddb3d9e1cea1 P aad0f35208db44fe8cf5fcc4b6a00a89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:48:55.686767 26192 log.cc:1196] T 1375c5d029404611a4e7ddb3d9e1cea1 P aad0f35208db44fe8cf5fcc4b6a00a89: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/wal/wals/1375c5d029404611a4e7ddb3d9e1cea1
I20260806 02:48:55.687088 26192 ts_tablet_manager.cc:1939] T 1375c5d029404611a4e7ddb3d9e1cea1 P aad0f35208db44fe8cf5fcc4b6a00a89: Deleting consensus metadata
I20260806 02:48:55.688482 25468 catalog_manager.cc:4841] TS aad0f35208db44fe8cf5fcc4b6a00a89 (127.1.183.131:42193): tablet 1375c5d029404611a4e7ddb3d9e1cea1 (table test-table [id=1ea4db1ded8f4b05839f2b9d8a74c476]) successfully deleted
I20260806 02:48:55.708614 26193 tablet_replica.cc:330] T 1375c5d029404611a4e7ddb3d9e1cea1 P 984378e530c349159d547c20dfef131f: stopping tablet replica
I20260806 02:48:55.709092 26193 raft_consensus.cc:2229] T 1375c5d029404611a4e7ddb3d9e1cea1 P 984378e530c349159d547c20dfef131f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:48:55.709522 26193 raft_consensus.cc:2258] T 1375c5d029404611a4e7ddb3d9e1cea1 P 984378e530c349159d547c20dfef131f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:48:55.711220 26193 ts_tablet_manager.cc:1905] T 1375c5d029404611a4e7ddb3d9e1cea1 P 984378e530c349159d547c20dfef131f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 02:48:55.712289 25467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.183.129:46301 (error 108)
I20260806 02:48:55.712805 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 25766
I20260806 02:48:55.716470 26165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:55.716035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:46301 (local address 127.0.0.1:58288)
0806 02:48:55.716255 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:55.716350 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.129:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260806 02:48:55.717650 25467 catalog_manager.cc:4584] TS d6dcb23d9e4f4ab49608bacef00f2a4b (127.1.183.129:46301): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1375c5d029404611a4e7ddb3d9e1cea1: Network error: Client connection negotiation failed: client connection to 127.1.183.129:46301: connect: Connection refused (error 111)
I20260806 02:48:55.724067 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 25971
I20260806 02:48:55.724216 26165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:48:55.723867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:45295 (local address 127.0.0.1:45940)
0806 02:48:55.724023 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 02:48:55.724085 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.130:45295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260806 02:48:55.750532 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 25442
I20260806 02:49:15.764415 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:34847
--webserver_interface=127.1.183.190
--webserver_port=44161
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:45809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:34847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:49:15.877120 26196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:15.877377 26196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:15.877415 26196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:15.881449 26196 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 02:49:15.881552 26196 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 02:49:15.881568 26196 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 02:49:15.881582 26196 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:49:15.881599 26196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:15.881666 26196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:15.881698 26196 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:49:15.881718 26196 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:49:15.886978 26196 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:45809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:34847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:34847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=44161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:15.888059 26196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:15.889166 26196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:49:15.897914 26196 server_base.cc:1034] running on GCE node
W20260806 02:49:15.898010 26206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:15.897944 26203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:15.898150 26202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:15.898706 26196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:15.899811 26196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:15.901005 26196 hybrid_clock.cc:648] HybridClock initialized: now 1785984555900973 us; error 41 us; skew 500 ppm
Aug 06 02:49:15 dist-test-slave-gmjp krb5kdc[24642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984555, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:15.903762 26196 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:49:15.904670 26196 webserver.cc:460] Webserver started at http://127.1.183.190:44161/ using document root <none> and password file <none>
I20260806 02:49:15.905215 26196 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:15.905329 26196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:15.909117 26196 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:49:15.911262 26220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:15.912616 26196 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 02:49:15.912760 26196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "3f14544cf1a246e7a63bdd29f16bec3d"
format_stamp: "Formatted at 2026-08-06 02:48:33 on dist-test-slave-gmjp"
server_key: "c7e94d71f5c498643f6a4bfe7ae67808"
server_key_iv: "f377a8d6ab6f19e5dcfdab60e8128a8e"
server_key_version: "encryptionkey@0"
I20260806 02:49:15.913331 26196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:15.930282 26196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:15.933362 26196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:15.933585 26196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:15.940946 26196 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:34847
I20260806 02:49:15.941242 26293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:34847 every 8 connection(s)
I20260806 02:49:15.942076 26196 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:49:15.946514 26294 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:49:15.948140 26294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d: Bootstrap starting.
I20260806 02:49:15.952374 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 26196
I20260806 02:49:15.952817 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.129:46301
--local_ip_for_outbound_sockets=127.1.183.129
--tserver_master_addrs=127.1.183.190:34847
--webserver_port=39921
--webserver_interface=127.1.183.129
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:45809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:49:15.966176 26294 log.cc:824] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:15.978251 26294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:49:15.978688 26294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d: Bootstrap complete.
I20260806 02:49:15.982867 26294 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f14544cf1a246e7a63bdd29f16bec3d" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34847 } }
I20260806 02:49:15.983291 26294 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f14544cf1a246e7a63bdd29f16bec3d, State: Initialized, Role: FOLLOWER
I20260806 02:49:15.983789 26294 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f14544cf1a246e7a63bdd29f16bec3d" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34847 } }
I20260806 02:49:15.983937 26294 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:49:15.984030 26294 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:49:15.984195 26294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [term 2 FOLLOWER]: Advancing to term 3
I20260806 02:49:15.986188 26294 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f14544cf1a246e7a63bdd29f16bec3d" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34847 } }
I20260806 02:49:15.986527 26294 leader_election.cc:302] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f14544cf1a246e7a63bdd29f16bec3d; no voters:
I20260806 02:49:15.986896 26294 leader_election.cc:288] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 02:49:15.987010 26304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [term 3 FOLLOWER]: Leader election won for term 3
I20260806 02:49:15.987222 26304 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [term 3 LEADER]: Becoming Leader. State: Replica: 3f14544cf1a246e7a63bdd29f16bec3d, State: Running, Role: LEADER
I20260806 02:49:15.987591 26304 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f14544cf1a246e7a63bdd29f16bec3d" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34847 } }
I20260806 02:49:15.988224 26294 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:49:15.989009 26305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3f14544cf1a246e7a63bdd29f16bec3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f14544cf1a246e7a63bdd29f16bec3d" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34847 } } }
I20260806 02:49:15.989144 26305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:15.989886 26306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f14544cf1a246e7a63bdd29f16bec3d. Latest consensus state: current_term: 3 leader_uuid: "3f14544cf1a246e7a63bdd29f16bec3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f14544cf1a246e7a63bdd29f16bec3d" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34847 } } }
I20260806 02:49:15.990001 26306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:15.990348 26313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:49:15.995469 26313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:49:15.996140 26313 catalog_manager.cc:1259] Loaded cluster ID: 066d508fdb7b43a180b5e3ae9155168f
I20260806 02:49:15.996224 26313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:49:15.998840 26313 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:49:15.999605 26313 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d: Loaded TSK: 0
I20260806 02:49:15.999691 26313 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3f14544cf1a246e7a63bdd29f16bec3d: Loaded TSK: 1
F20260806 02:49:16.001036 26313 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff4750049ad google::LogMessage::Fail() at ??:0
@ 0x7ff47500788a google::LogMessage::SendToLog() at ??:0
@ 0x7ff4750044de google::LogMessage::Flush() at ??:0
@ 0x7ff475005a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff477cc818d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ff477cc0f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7ff47f9697bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7ff47f94399c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7ff47f944178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7ff47f985df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7ff47f9a482b std::function<>::operator()() at ??:0
@ 0x7ff47f943b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7ff47f944f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff47f941579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7ff47f985640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ff4772563d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff477256c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7ff4772583f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ff477247512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff47888b6db start_thread at ??:0
@ 0x7ff473b0261f clone at ??:0
W20260806 02:49:16.105978 26296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:16.106268 26296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:16.106316 26296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:16.110532 26296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:16.110600 26296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:16.110695 26296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.129
I20260806 02:49:16.116371 26296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:45809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.129:46301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.183.129
--webserver_port=39921
--tserver_master_addrs=127.1.183.190:34847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.129
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:16.117565 26296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:16.118705 26296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:16.129675 26365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:16.129863 26371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:16.130128 26368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:16.130780 26296 server_base.cc:1034] running on GCE node
I20260806 02:49:16.131249 26296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:16.131968 26296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:16.133188 26296 hybrid_clock.cc:648] HybridClock initialized: now 1785984556133145 us; error 59 us; skew 500 ppm
Aug 06 02:49:16 dist-test-slave-gmjp krb5kdc[24642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984556, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:16.136270 26296 init.cc:377] Logged in from keytab as kudu/127.1.183.129@KRBTEST.COM (short username kudu)
I20260806 02:49:16.137315 26296 webserver.cc:460] Webserver started at http://127.1.183.129:39921/ using document root <none> and password file <none>
I20260806 02:49:16.137925 26296 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:16.138026 26296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:16.142252 26296 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:49:16.144609 26383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:16.145996 26296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:49:16.146096 26296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/wal
uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b"
format_stamp: "Formatted at 2026-08-06 02:48:33 on dist-test-slave-gmjp"
server_key: "a6c81690ae02b271667092f1e57e6cd0"
server_key_iv: "fac9fe9d154cb5733cde54b2b84eca91"
server_key_version: "encryptionkey@0"
I20260806 02:49:16.146679 26296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:16.161338 26296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:16.164759 26296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:16.164968 26296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:16.165642 26296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:49:16.166785 26392 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 02:49:16.171166 26296 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 02:49:16.171231 26296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260806 02:49:16.171278 26296 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 02:49:16.173069 26296 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 02:49:16.173127 26296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260806 02:49:16.173388 26392 tablet_bootstrap.cc:492] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b: Bootstrap starting.
I20260806 02:49:16.180577 26392 log.cc:824] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:16.183807 26296 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.129:46301
I20260806 02:49:16.184043 26568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.129:46301 every 8 connection(s)
I20260806 02:49:16.184897 26296 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
I20260806 02:49:16.189836 26392 tablet_bootstrap.cc:492] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:49:16.190243 26392 tablet_bootstrap.cc:492] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b: Bootstrap complete.
I20260806 02:49:16.190580 26392 ts_tablet_manager.cc:1397] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
I20260806 02:49:16.191452 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 26296
I20260806 02:49:16.191948 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.130:45295
--local_ip_for_outbound_sockets=127.1.183.130
--tserver_master_addrs=127.1.183.190:34847
--webserver_port=46265
--webserver_interface=127.1.183.130
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:45809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:49:16.193090 26392 raft_consensus.cc:348] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984378e530c349159d547c20dfef131f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 45295 } } peers { permanent_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 42193 } } peers { permanent_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46301 } }
I20260806 02:49:16.193511 26392 raft_consensus.cc:729] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6dcb23d9e4f4ab49608bacef00f2a4b, State: Initialized, Role: FOLLOWER
I20260806 02:49:16.194047 26392 consensus_queue.cc:260] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984378e530c349159d547c20dfef131f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 45295 } } peers { permanent_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 42193 } } peers { permanent_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46301 } }
I20260806 02:49:16.195147 26392 ts_tablet_manager.cc:1428] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260806 02:49:16.228039 26571 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:49:16.187067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.129:40455)
0806 02:49:16.187663 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:16.187722 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:16.188828 (+ 1106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:16.227164 (+ 38336us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":170,"threads_started":1}
W20260806 02:49:16.228711 26569 heartbeater.cc:644] Failed to heartbeat to 127.1.183.190:34847 (0 consecutive failures): Network error: Failed to ping master at 127.1.183.190:34847: Client connection negotiation failed: client connection to 127.1.183.190:34847: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 02:49:16.229450 26571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:16.229178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.129:47165)
0806 02:49:16.229309 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:16.229378 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260806 02:49:16.230156 26571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:16.229962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.129:43635)
0806 02:49:16.230062 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:16.230112 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 02:49:16.230310 26569 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:49:16.314465 26573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:16.314781 26573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:16.314858 26573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:16.318953 26573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:16.319047 26573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:16.319141 26573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.130
I20260806 02:49:16.325914 26573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:45809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.130:45295
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.183.130
--webserver_port=46265
--tserver_master_addrs=127.1.183.190:34847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.130
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:16.327011 26573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:16.328048 26573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:16.337419 26584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:16.337466 26583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:16.337414 26587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:16.337615 26573 server_base.cc:1034] running on GCE node
I20260806 02:49:16.338141 26573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:16.338786 26573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:16.339980 26573 hybrid_clock.cc:648] HybridClock initialized: now 1785984556339970 us; error 30 us; skew 500 ppm
Aug 06 02:49:16 dist-test-slave-gmjp krb5kdc[24642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984556, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:16.342684 26573 init.cc:377] Logged in from keytab as kudu/127.1.183.130@KRBTEST.COM (short username kudu)
I20260806 02:49:16.343515 26573 webserver.cc:460] Webserver started at http://127.1.183.130:46265/ using document root <none> and password file <none>
I20260806 02:49:16.344084 26573 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:16.344198 26573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:16.347882 26573 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260806 02:49:16.350349 26596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:16.351661 26573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 02:49:16.351800 26573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/wal
uuid: "984378e530c349159d547c20dfef131f"
format_stamp: "Formatted at 2026-08-06 02:48:33 on dist-test-slave-gmjp"
server_key: "80db4ce4d4cd2f3544e1f74b38c73a38"
server_key_iv: "c0558eb2182cb3d003d28da14c166b42"
server_key_version: "encryptionkey@0"
I20260806 02:49:16.352439 26573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:16.365782 26573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:16.369302 26573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:16.369474 26573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:16.370039 26573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:49:16.371093 26608 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260806 02:49:16.374904 26608 ts_tablet_manager.cc:1852] T 1375c5d029404611a4e7ddb3d9e1cea1 P 984378e530c349159d547c20dfef131f: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260806 02:49:16.375300 26608 ts_tablet_manager.cc:1905] T 1375c5d029404611a4e7ddb3d9e1cea1 P 984378e530c349159d547c20dfef131f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:49:16.379314 26608 ts_tablet_manager.cc:1918] T 1375c5d029404611a4e7ddb3d9e1cea1 P 984378e530c349159d547c20dfef131f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:49:16.379410 26608 log.cc:1196] T 1375c5d029404611a4e7ddb3d9e1cea1 P 984378e530c349159d547c20dfef131f: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/wal/wals/1375c5d029404611a4e7ddb3d9e1cea1
I20260806 02:49:16.379777 26608 ts_tablet_manager.cc:1939] T 1375c5d029404611a4e7ddb3d9e1cea1 P 984378e530c349159d547c20dfef131f: Deleting consensus metadata
I20260806 02:49:16.380187 26573 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260806 02:49:16.380256 26573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260806 02:49:16.380332 26573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:49:16.380365 26573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:16.389250 26573 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.130:45295
I20260806 02:49:16.389336 26765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.130:45295 every 8 connection(s)
I20260806 02:49:16.390138 26573 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
I20260806 02:49:16.390542 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 26573
I20260806 02:49:16.391047 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.131:42193
--local_ip_for_outbound_sockets=127.1.183.131
--tserver_master_addrs=127.1.183.190:34847
--webserver_port=33283
--webserver_interface=127.1.183.131
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:45809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:49:16.394155 26772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:16.392765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.130:37729)
0806 02:49:16.393280 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:16.393411 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":143,"threads_started":1}
W20260806 02:49:16.394642 26767 heartbeater.cc:644] Failed to heartbeat to 127.1.183.190:34847 (0 consecutive failures): Network error: Failed to ping master at 127.1.183.190:34847: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
I20260806 02:49:16.395320 26772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:16.395097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.130:49025)
0806 02:49:16.395217 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:16.395266 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 02:49:16.395953 26772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:16.395731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.130:46735)
0806 02:49:16.395838 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:16.395886 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 02:49:16.396112 26767 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:49:16.507494 26770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:16.507813 26770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:16.507890 26770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:16.512013 26770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:16.512115 26770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:16.512235 26770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.131
I20260806 02:49:16.517547 26770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:45809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.131:42193
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.183.131
--webserver_port=33283
--tserver_master_addrs=127.1.183.190:34847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.131
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:16.518641 26770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:16.519678 26770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:16.529340 26783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:16.529340 26778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:16.529347 26779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:16.529882 26770 server_base.cc:1034] running on GCE node
I20260806 02:49:16.530314 26770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:16.530989 26770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:16.532217 26770 hybrid_clock.cc:648] HybridClock initialized: now 1785984556532200 us; error 27 us; skew 500 ppm
Aug 06 02:49:16 dist-test-slave-gmjp krb5kdc[24642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984556, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:16.534993 26770 init.cc:377] Logged in from keytab as kudu/127.1.183.131@KRBTEST.COM (short username kudu)
I20260806 02:49:16.535897 26770 webserver.cc:460] Webserver started at http://127.1.183.131:33283/ using document root <none> and password file <none>
I20260806 02:49:16.536563 26770 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:16.536659 26770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:16.540484 26770 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:49:16.542569 26797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:16.543740 26770 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:49:16.543874 26770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/wal
uuid: "aad0f35208db44fe8cf5fcc4b6a00a89"
format_stamp: "Formatted at 2026-08-06 02:48:34 on dist-test-slave-gmjp"
server_key: "c9be08837258de05c57d727320032012"
server_key_iv: "a6ce26e6205e3eec22eec21ad4c27bbe"
server_key_version: "encryptionkey@0"
I20260806 02:49:16.544477 26770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:16.555538 26770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:16.558421 26770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:16.558629 26770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:16.559201 26770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:49:16.560019 26770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:49:16.560091 26770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:16.560184 26770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:49:16.560230 26770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:16.570009 26770 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.131:42193
I20260806 02:49:16.570042 26968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.131:42193 every 8 connection(s)
I20260806 02:49:16.571038 26770 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
I20260806 02:49:16.574630 26977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:16.573116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.131:55505)
0806 02:49:16.573805 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:16.573911 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":116,"threads_started":1}
W20260806 02:49:16.575060 26971 heartbeater.cc:644] Failed to heartbeat to 127.1.183.190:34847 (0 consecutive failures): Network error: Failed to ping master at 127.1.183.190:34847: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
I20260806 02:49:16.575651 26977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:16.575442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.131:47393)
0806 02:49:16.575549 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:16.575597 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260806 02:49:16.576285 26977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:16.576054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.131:55755)
0806 02:49:16.576144 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:16.576238 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260806 02:49:16.576459 26971 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 02:49:16.579207 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 26770
I20260806 02:49:17.231298 26979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:17.230865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.129:33241)
0806 02:49:17.231137 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:17.231196 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":80,"threads_started":1}
I20260806 02:49:17.323657 26981 raft_consensus.cc:482] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:49:17.323891 26981 raft_consensus.cc:504] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984378e530c349159d547c20dfef131f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 45295 } } peers { permanent_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 42193 } } peers { permanent_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46301 } }
I20260806 02:49:17.325083 26981 leader_election.cc:288] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 984378e530c349159d547c20dfef131f (127.1.183.130:45295), aad0f35208db44fe8cf5fcc4b6a00a89 (127.1.183.131:42193)
Aug 06 02:49:17 dist-test-slave-gmjp krb5kdc[24642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984556, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for kudu/127.1.183.130@KRBTEST.COM
Aug 06 02:49:17 dist-test-slave-gmjp krb5kdc[24642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984556, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for kudu/127.1.183.131@KRBTEST.COM
I20260806 02:49:17.342854 26979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:17.325316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:45295 (local address 127.1.183.129:60423)
0806 02:49:17.325475 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:17.325491 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:17.325606 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:17.327699 (+ 2093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:17.327704 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:17.328352 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:49:17.328903 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:17.328918 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:17.330189 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:17.330192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:17.331247 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:17.331257 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:17.331977 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:17.331981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:17.332270 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:17.335928 (+ 3658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:49:17.335967 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:49:17.337928 (+ 1961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:49:17.340231 (+ 2303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:17.340239 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:17.340255 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:17.340676 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:17.341055 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:17.341061 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:17.341065 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:17.341233 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:17.341861 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:49:17.341864 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:49:17.342102 (+ 238us) client_negotiation.cc:769] Sending connection context
0806 02:49:17.342287 (+ 185us) client_negotiation.cc:240] Negotiation successful
0806 02:49:17.342636 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":267,"spinlock_wait_cycles":63360}
I20260806 02:49:17.342854 26984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:17.325543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:44475 (local address 127.1.183.131:42193)
0806 02:49:17.326062 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:17.326073 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:17.326105 (+ 32us) server_negotiation.cc:407] Connection header received
0806 02:49:17.326835 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:17.326843 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:17.327329 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:17.327647 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:17.329081 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:17.330034 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:17.331374 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:17.331839 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:17.332030 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:17.338341 (+ 6311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:17.338346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:17.338358 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:17.338421 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:17.340377 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:17.340861 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:17.340866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:17.340870 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:17.340987 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:17.341323 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:17.341329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:17.341331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:17.341732 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:17.341878 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:17.342411 (+ 533us) server_negotiation.cc:299] Negotiation successful
0806 02:49:17.342674 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":151,"threads_started":1}
I20260806 02:49:17.342896 26982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:17.325332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:42193 (local address 127.1.183.129:44475)
0806 02:49:17.325819 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:17.325833 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:17.325936 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:17.327697 (+ 1761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:17.327703 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:17.328354 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:49:17.328903 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:17.328918 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:17.330195 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:17.330198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:17.331245 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:17.331256 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:17.332000 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:17.332004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:17.332230 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:17.335962 (+ 3732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:49:17.335979 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:49:17.338146 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:49:17.340496 (+ 2350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:17.340499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:17.340500 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:17.340748 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:17.341109 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:17.341111 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:17.341113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:17.341223 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:17.341856 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:49:17.341862 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:49:17.342100 (+ 238us) client_negotiation.cc:769] Sending connection context
0806 02:49:17.342288 (+ 188us) client_negotiation.cc:240] Negotiation successful
0806 02:49:17.342723 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":377,"spinlock_wait_cycles":18560,"thread_start_us":73,"threads_started":1}
I20260806 02:49:17.342890 26983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:17.325587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:60423 (local address 127.1.183.130:45295)
0806 02:49:17.325894 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:17.325908 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:17.325949 (+ 41us) server_negotiation.cc:407] Connection header received
0806 02:49:17.326898 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:17.326908 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:17.327330 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:17.327647 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:17.329088 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:17.330028 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:17.331372 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:17.331839 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:17.332016 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:17.338271 (+ 6255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:17.338277 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:17.338289 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:17.338318 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:17.340080 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:17.340796 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:17.340805 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:17.340813 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:17.340952 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:17.341365 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:17.341368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:17.341369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:17.341732 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:17.341878 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:17.342418 (+ 540us) server_negotiation.cc:299] Negotiation successful
0806 02:49:17.342701 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":85,"threads_started":1}
W20260806 02:49:17.345438 26385 leader_election.cc:341] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aad0f35208db44fe8cf5fcc4b6a00a89 (127.1.183.131:42193): Not found: Tablet not found: 1375c5d029404611a4e7ddb3d9e1cea1
W20260806 02:49:17.345611 26386 leader_election.cc:341] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 984378e530c349159d547c20dfef131f (127.1.183.130:45295): Not found: Tablet not found: 1375c5d029404611a4e7ddb3d9e1cea1
I20260806 02:49:17.345688 26386 leader_election.cc:302] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6dcb23d9e4f4ab49608bacef00f2a4b; no voters: 984378e530c349159d547c20dfef131f, aad0f35208db44fe8cf5fcc4b6a00a89
I20260806 02:49:17.345950 26981 raft_consensus.cc:2726] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:49:17.397040 26985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:17.396666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.130:41863)
0806 02:49:17.396895 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:17.396956 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":72,"threads_started":1}
I20260806 02:49:17.577418 26987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:17.576974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.131:56501)
0806 02:49:17.577245 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:17.577304 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":89,"threads_started":1}
I20260806 02:49:18.232730 26989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:18.232243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.129:57805)
0806 02:49:18.232571 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:18.232629 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":104,"threads_started":1}
I20260806 02:49:18.398272 26992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:18.397843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.130:51211)
0806 02:49:18.398111 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:18.398174 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":82,"threads_started":1}
I20260806 02:49:18.578673 26995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:18.578204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.131:51399)
0806 02:49:18.578497 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:18.578561 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":75,"threads_started":1}
I20260806 02:49:18.883983 26996 raft_consensus.cc:482] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:49:18.884119 26996 raft_consensus.cc:504] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984378e530c349159d547c20dfef131f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 45295 } } peers { permanent_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 42193 } } peers { permanent_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46301 } }
I20260806 02:49:18.884657 26996 leader_election.cc:288] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 984378e530c349159d547c20dfef131f (127.1.183.130:45295), aad0f35208db44fe8cf5fcc4b6a00a89 (127.1.183.131:42193)
W20260806 02:49:18.885592 26386 leader_election.cc:341] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 984378e530c349159d547c20dfef131f (127.1.183.130:45295): Not found: Tablet not found: 1375c5d029404611a4e7ddb3d9e1cea1
W20260806 02:49:18.885696 26385 leader_election.cc:341] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aad0f35208db44fe8cf5fcc4b6a00a89 (127.1.183.131:42193): Not found: Tablet not found: 1375c5d029404611a4e7ddb3d9e1cea1
I20260806 02:49:18.885768 26385 leader_election.cc:302] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6dcb23d9e4f4ab49608bacef00f2a4b; no voters: 984378e530c349159d547c20dfef131f, aad0f35208db44fe8cf5fcc4b6a00a89
I20260806 02:49:18.885900 26996 raft_consensus.cc:2726] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:49:19.234190 26997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:19.233686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.129:54473)
0806 02:49:19.234023 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:19.234089 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":107,"threads_started":1}
I20260806 02:49:19.399693 26998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:19.399202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.130:57859)
0806 02:49:19.399520 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:19.399592 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":77,"threads_started":1}
I20260806 02:49:19.579911 26999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:19.579489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.131:42153)
0806 02:49:19.579754 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:19.579815 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":76,"threads_started":1}
I20260806 02:49:20.235455 27000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:20.234988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.129:46411)
0806 02:49:20.235296 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:20.235356 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":78,"threads_started":1}
I20260806 02:49:20.401182 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:20.400725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.130:43225)
0806 02:49:20.401018 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:20.401082 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":86,"threads_started":1}
I20260806 02:49:20.581153 27004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:20.580735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.131:51113)
0806 02:49:20.581000 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:20.581054 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":78,"threads_started":1}
I20260806 02:49:21.236979 27005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:21.236500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.129:59121)
0806 02:49:21.236817 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:21.236879 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":89,"threads_started":1}
I20260806 02:49:21.402536 27008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:21.402055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.130:46255)
0806 02:49:21.402350 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:21.402417 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":80,"threads_started":1}
I20260806 02:49:21.533238 27010 raft_consensus.cc:482] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:49:21.533442 27010 raft_consensus.cc:504] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984378e530c349159d547c20dfef131f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 45295 } } peers { permanent_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 42193 } } peers { permanent_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46301 } }
I20260806 02:49:21.533977 27010 leader_election.cc:288] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 984378e530c349159d547c20dfef131f (127.1.183.130:45295), aad0f35208db44fe8cf5fcc4b6a00a89 (127.1.183.131:42193)
W20260806 02:49:21.534925 26385 leader_election.cc:341] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aad0f35208db44fe8cf5fcc4b6a00a89 (127.1.183.131:42193): Not found: Tablet not found: 1375c5d029404611a4e7ddb3d9e1cea1
W20260806 02:49:21.535038 26386 leader_election.cc:341] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 984378e530c349159d547c20dfef131f (127.1.183.130:45295): Not found: Tablet not found: 1375c5d029404611a4e7ddb3d9e1cea1
I20260806 02:49:21.535161 26386 leader_election.cc:302] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6dcb23d9e4f4ab49608bacef00f2a4b; no voters: 984378e530c349159d547c20dfef131f, aad0f35208db44fe8cf5fcc4b6a00a89
I20260806 02:49:21.535302 27010 raft_consensus.cc:2726] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:49:21.582337 27011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:21.581919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.131:38129)
0806 02:49:21.582186 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:21.582242 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":101,"threads_started":1}
I20260806 02:49:22.238159 27014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:22.237741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.129:54919)
0806 02:49:22.238000 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:22.238059 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":80,"threads_started":1}
I20260806 02:49:22.403757 27015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:22.403315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.130:37867)
0806 02:49:22.403597 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:22.403659 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":76,"threads_started":1}
I20260806 02:49:22.583792 27017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:22.583294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.131:58345)
0806 02:49:22.583621 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:22.583688 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":83,"threads_started":1}
I20260806 02:49:23.239418 27019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:23.238980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.129:52491)
0806 02:49:23.239262 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:23.239316 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":106,"threads_started":1}
I20260806 02:49:23.405162 27020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:23.404642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.130:46383)
0806 02:49:23.404990 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:23.405056 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":77,"threads_started":1}
I20260806 02:49:23.585196 27021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:23.584613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.131:32871)
0806 02:49:23.584992 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:23.585067 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":91,"threads_started":1}
I20260806 02:49:24.240759 27022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:24.240306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.129:53369)
0806 02:49:24.240592 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:24.240654 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":78,"threads_started":1}
I20260806 02:49:24.406458 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:24.405972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.130:56639)
0806 02:49:24.406280 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:24.406349 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":100,"threads_started":1}
I20260806 02:49:24.586318 27027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:24.585904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.131:35963)
0806 02:49:24.586164 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:24.586221 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":75,"threads_started":1}
I20260806 02:49:25.242003 27028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:25.241579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.129:33979)
0806 02:49:25.241842 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:25.241899 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":80,"threads_started":1}
I20260806 02:49:25.407768 27029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:25.407320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.130:41579)
0806 02:49:25.407605 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:25.407663 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":97,"threads_started":1}
I20260806 02:49:25.587690 27030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:25.587224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.131:43587)
0806 02:49:25.587527 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:25.587587 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":100,"threads_started":1}
I20260806 02:49:25.885125 27031 raft_consensus.cc:482] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:49:25.885304 27031 raft_consensus.cc:504] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984378e530c349159d547c20dfef131f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 45295 } } peers { permanent_uuid: "aad0f35208db44fe8cf5fcc4b6a00a89" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 42193 } } peers { permanent_uuid: "d6dcb23d9e4f4ab49608bacef00f2a4b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 46301 } }
I20260806 02:49:25.885815 27031 leader_election.cc:288] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 984378e530c349159d547c20dfef131f (127.1.183.130:45295), aad0f35208db44fe8cf5fcc4b6a00a89 (127.1.183.131:42193)
W20260806 02:49:25.886613 26385 leader_election.cc:341] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aad0f35208db44fe8cf5fcc4b6a00a89 (127.1.183.131:42193): Not found: Tablet not found: 1375c5d029404611a4e7ddb3d9e1cea1
W20260806 02:49:25.886719 26386 leader_election.cc:341] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 984378e530c349159d547c20dfef131f (127.1.183.130:45295): Not found: Tablet not found: 1375c5d029404611a4e7ddb3d9e1cea1
I20260806 02:49:25.886759 26385 leader_election.cc:302] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6dcb23d9e4f4ab49608bacef00f2a4b; no voters: 984378e530c349159d547c20dfef131f, aad0f35208db44fe8cf5fcc4b6a00a89
I20260806 02:49:25.886907 27031 raft_consensus.cc:2726] T 1375c5d029404611a4e7ddb3d9e1cea1 P d6dcb23d9e4f4ab49608bacef00f2a4b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:49:26.243454 27032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:26.242998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.129:60349)
0806 02:49:26.243287 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:26.243347 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":75,"threads_started":1}
I20260806 02:49:26.409222 27033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:26.408767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.130:46101)
0806 02:49:26.409055 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:26.409122 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":80,"threads_started":1}
I20260806 02:49:26.588892 27034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:26.588447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.131:47841)
0806 02:49:26.588706 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:26.588781 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":89,"threads_started":1}
I20260806 02:49:27.244714 27035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:27.244313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.129:43647)
0806 02:49:27.244558 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:27.244614 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":79,"threads_started":1}
I20260806 02:49:27.410421 27037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:27.409995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.130:50015)
0806 02:49:27.410261 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:27.410321 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":79,"threads_started":1}
I20260806 02:49:27.590041 27039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:27.589616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.131:58401)
0806 02:49:27.589887 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:27.589939 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":98,"threads_started":1}
I20260806 02:49:28.245961 27040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:28.245532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.129:37079)
0806 02:49:28.245793 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:28.245849 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":78,"threads_started":1}
I20260806 02:49:28.411906 27041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:28.411439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.130:42279)
0806 02:49:28.411727 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:28.411796 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":81,"threads_started":1}
I20260806 02:49:28.591332 27043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:28.590894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.131:47027)
0806 02:49:28.591166 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:28.591216 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":107,"threads_started":1}
I20260806 02:49:29.247287 27045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:29.246826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.129:56171)
0806 02:49:29.247124 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:29.247187 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":107,"threads_started":1}
I20260806 02:49:29.413271 27046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:29.412808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.130:39253)
0806 02:49:29.413102 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:29.413168 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":78,"threads_started":1}
I20260806 02:49:29.592573 27047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:29.592109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.131:40057)
0806 02:49:29.592418 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:29.592473 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":146,"threads_started":1}
I20260806 02:49:30.248569 27048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:30.248046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.129:44257)
0806 02:49:30.248386 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:30.248455 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":123,"threads_started":1}
I20260806 02:49:30.414554 27049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:30.414102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.130:48671)
0806 02:49:30.414397 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:30.414455 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":79,"threads_started":1}
I20260806 02:49:30.593848 27052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:30.593352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.131:35147)
0806 02:49:30.593676 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:30.593745 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":79,"threads_started":1}
I20260806 02:49:31.250043 27053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:31.249579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.129:48589)
0806 02:49:31.249863 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:31.249942 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":69,"threads_started":1}
I20260806 02:49:31.415982 27054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:31.415554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.130:51451)
0806 02:49:31.415824 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:31.415883 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":75,"threads_started":1}
I20260806 02:49:31.581277 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 26296
I20260806 02:49:31.592731 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 26573
I20260806 02:49:31.597855 27055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:31.597441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34847 (local address 127.1.183.131:33103)
0806 02:49:31.597707 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:31.597761 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.190:34847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":90,"threads_started":1}
I20260806 02:49:31.602420 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 26770
2026-08-06T02:49:31Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59934 ms)
[----------] 30 tests from SecurityITest (257713 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:49:31 dist-test-slave-gmjp krb5kdc[27058](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:49:31 dist-test-slave-gmjp krb5kdc[27058](info): set up 2 sockets
Aug 06 02:49:31 dist-test-slave-gmjp krb5kdc[27058](info): commencing operation
krb5kdc: starting...
W20260806 02:49:32.855495 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.226s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:49:32 dist-test-slave-gmjp krb5kdc[27058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:49:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:49:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
I20260806 02:49:32.984709 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:34565
--webserver_interface=127.1.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:43967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:34565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:49:33.099428 27078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:33.099748 27078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:33.099823 27078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:33.103785 27078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:49:33.103881 27078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:33.103917 27078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:33.103960 27078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:49:33.103994 27078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:49:33.109941 27078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:43967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:34565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:34565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:33.111052 27078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:33.112126 27078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:33.120908 27088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:33.120967 27078 server_base.cc:1034] running on GCE node
W20260806 02:49:33.120929 27084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:33.120926 27086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:33.121809 27078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:33.122895 27078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:33.124063 27078 hybrid_clock.cc:648] HybridClock initialized: now 1785984573124043 us; error 30 us; skew 500 ppm
Aug 06 02:49:33 dist-test-slave-gmjp krb5kdc[27058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984573, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:33.126875 27078 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:49:33.127770 27078 webserver.cc:460] Webserver started at http://127.1.183.190:34145/ using document root <none> and password file <none>
I20260806 02:49:33.128371 27078 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:33.128435 27078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:33.128641 27078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:33.131121 27078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "965476a1bbf349ed998400519b74356c"
format_stamp: "Formatted at 2026-08-06 02:49:33 on dist-test-slave-gmjp"
server_key: "5e2bc6511a0b38e0efc5cc9de2c6ec8f"
server_key_iv: "d61c7fc01307d0c28829d30828821adf"
server_key_version: "encryptionkey@0"
I20260806 02:49:33.131664 27078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "965476a1bbf349ed998400519b74356c"
format_stamp: "Formatted at 2026-08-06 02:49:33 on dist-test-slave-gmjp"
server_key: "5e2bc6511a0b38e0efc5cc9de2c6ec8f"
server_key_iv: "d61c7fc01307d0c28829d30828821adf"
server_key_version: "encryptionkey@0"
I20260806 02:49:33.135439 27078 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:49:33.137852 27095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:33.139125 27078 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:49:33.139257 27078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "965476a1bbf349ed998400519b74356c"
format_stamp: "Formatted at 2026-08-06 02:49:33 on dist-test-slave-gmjp"
server_key: "5e2bc6511a0b38e0efc5cc9de2c6ec8f"
server_key_iv: "d61c7fc01307d0c28829d30828821adf"
server_key_version: "encryptionkey@0"
I20260806 02:49:33.139364 27078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:33.152346 27078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:33.155305 27078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:33.155517 27078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:33.163262 27078 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:34565
I20260806 02:49:33.163288 27176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:34565 every 8 connection(s)
I20260806 02:49:33.164350 27078 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:49:33.167107 27177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:33.171566 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 27078
I20260806 02:49:33.171706 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:49:33.172057 1758 external_mini_cluster.cc:1428] Setting key 7401ec7b302112cac5efe6b7c8ecc6a5
I20260806 02:49:33.173214 27177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 965476a1bbf349ed998400519b74356c: Bootstrap starting.
I20260806 02:49:33.175539 27177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 965476a1bbf349ed998400519b74356c: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:33.176344 27177 log.cc:824] T 00000000000000000000000000000000 P 965476a1bbf349ed998400519b74356c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:33.178263 27177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 965476a1bbf349ed998400519b74356c: No bootstrap required, opened a new log
Aug 06 02:49:33 dist-test-slave-gmjp krb5kdc[27058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:49:33.182972 27177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 965476a1bbf349ed998400519b74356c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965476a1bbf349ed998400519b74356c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34565 } }
I20260806 02:49:33.183286 27177 raft_consensus.cc:374] T 00000000000000000000000000000000 P 965476a1bbf349ed998400519b74356c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:33.183384 27177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 965476a1bbf349ed998400519b74356c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 965476a1bbf349ed998400519b74356c, State: Initialized, Role: FOLLOWER
I20260806 02:49:33.183943 27177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 965476a1bbf349ed998400519b74356c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965476a1bbf349ed998400519b74356c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34565 } }
I20260806 02:49:33.184175 27177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 965476a1bbf349ed998400519b74356c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:49:33.184273 27177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 965476a1bbf349ed998400519b74356c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:49:33.184384 27177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 965476a1bbf349ed998400519b74356c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:33.185678 27177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 965476a1bbf349ed998400519b74356c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965476a1bbf349ed998400519b74356c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34565 } }
I20260806 02:49:33.186079 27177 leader_election.cc:302] T 00000000000000000000000000000000 P 965476a1bbf349ed998400519b74356c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 965476a1bbf349ed998400519b74356c; no voters:
I20260806 02:49:33.186456 27177 leader_election.cc:288] T 00000000000000000000000000000000 P 965476a1bbf349ed998400519b74356c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:49:33.186622 27185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 965476a1bbf349ed998400519b74356c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:49:33.186920 27185 raft_consensus.cc:686] T 00000000000000000000000000000000 P 965476a1bbf349ed998400519b74356c [term 1 LEADER]: Becoming Leader. State: Replica: 965476a1bbf349ed998400519b74356c, State: Running, Role: LEADER
I20260806 02:49:33.187302 27185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 965476a1bbf349ed998400519b74356c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965476a1bbf349ed998400519b74356c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34565 } }
I20260806 02:49:33.187866 27177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 965476a1bbf349ed998400519b74356c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:49:33.189215 27188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 965476a1bbf349ed998400519b74356c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "965476a1bbf349ed998400519b74356c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965476a1bbf349ed998400519b74356c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34565 } } }
I20260806 02:49:33.189343 27188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 965476a1bbf349ed998400519b74356c [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:33.189530 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:33.173579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58290 (local address 127.1.183.190:34565)
0806 02:49:33.174007 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:33.174022 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:33.174057 (+ 35us) server_negotiation.cc:407] Connection header received
0806 02:49:33.174682 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:33.174698 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:33.175030 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:33.175368 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:33.176225 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:33.177461 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:33.178349 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:33.178826 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:33.178997 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:33.182097 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:33.182104 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:33.182123 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:33.182155 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:33.184840 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:33.185463 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:33.185472 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:33.185479 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:33.185637 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:33.185978 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:33.185981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:33.185983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:33.186610 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:33.186812 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:33.188088 (+ 1276us) server_negotiation.cc:299] Negotiation successful
0806 02:49:33.188431 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":121,"threads_started":1}
I20260806 02:49:33.189585 27188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 965476a1bbf349ed998400519b74356c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 965476a1bbf349ed998400519b74356c. Latest consensus state: current_term: 1 leader_uuid: "965476a1bbf349ed998400519b74356c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "965476a1bbf349ed998400519b74356c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34565 } } }
I20260806 02:49:33.189666 27188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 965476a1bbf349ed998400519b74356c [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:33.191078 27197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:49:33.194645 27197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:49:33.200706 27197 catalog_manager.cc:1347] Generated new cluster ID: 290a617fbb51458abc53f56f4bc53891
I20260806 02:49:33.200847 27197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:49:33.212575 27197 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:49:33.213783 27197 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:49:33.221450 27197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 965476a1bbf349ed998400519b74356c: Generated new TSK 0
I20260806 02:49:33.222422 27197 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
I20260806 02:49:33.259001 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.129:0
--local_ip_for_outbound_sockets=127.1.183.129
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:34565
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:43967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:49:33.376739 27212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:33.377012 27212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:33.377053 27212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:33.381078 27212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:33.381140 27212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:33.381201 27212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.129
I20260806 02:49:33.386562 27212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:43967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:34565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.129
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:33.387635 27212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:33.388895 27212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:33.399114 27218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:33.399106 27217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:33.399106 27220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:33.399477 27212 server_base.cc:1034] running on GCE node
I20260806 02:49:33.399968 27212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:33.400720 27212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:33.401913 27212 hybrid_clock.cc:648] HybridClock initialized: now 1785984573401897 us; error 28 us; skew 500 ppm
Aug 06 02:49:33 dist-test-slave-gmjp krb5kdc[27058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984573, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:33.404824 27212 init.cc:377] Logged in from keytab as kudu/127.1.183.129@KRBTEST.COM (short username kudu)
I20260806 02:49:33.405750 27212 webserver.cc:460] Webserver started at http://127.1.183.129:34151/ using document root <none> and password file <none>
I20260806 02:49:33.406342 27212 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:33.406445 27212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:33.406683 27212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:33.408865 27212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-0/data/instance:
uuid: "757473c25e6848909ab475b49bede65d"
format_stamp: "Formatted at 2026-08-06 02:49:33 on dist-test-slave-gmjp"
server_key: "f5606875cc426fe318a7f2557c0780d2"
server_key_iv: "49f4519567fb1bc82490433a12850e16"
server_key_version: "encryptionkey@0"
I20260806 02:49:33.409449 27212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance:
uuid: "757473c25e6848909ab475b49bede65d"
format_stamp: "Formatted at 2026-08-06 02:49:33 on dist-test-slave-gmjp"
server_key: "f5606875cc426fe318a7f2557c0780d2"
server_key_iv: "49f4519567fb1bc82490433a12850e16"
server_key_version: "encryptionkey@0"
I20260806 02:49:33.413336 27212 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260806 02:49:33.415896 27229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:33.417205 27212 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:49:33.417366 27212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-0/wal
uuid: "757473c25e6848909ab475b49bede65d"
format_stamp: "Formatted at 2026-08-06 02:49:33 on dist-test-slave-gmjp"
server_key: "f5606875cc426fe318a7f2557c0780d2"
server_key_iv: "49f4519567fb1bc82490433a12850e16"
server_key_version: "encryptionkey@0"
I20260806 02:49:33.417495 27212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:33.430797 27212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:33.433920 27212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:33.434128 27212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:33.434741 27212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:49:33.435660 27212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:49:33.435727 27212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:33.435801 27212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:49:33.435850 27212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:33.446722 27212 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.129:32781
I20260806 02:49:33.446739 27394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.129:32781 every 8 connection(s)
I20260806 02:49:33.447713 27212 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
I20260806 02:49:33.456050 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 27212
I20260806 02:49:33.456243 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance
I20260806 02:49:33.456593 1758 external_mini_cluster.cc:1428] Setting key df4a425fe66845c9328dd87f562daaf8
Aug 06 02:49:33 dist-test-slave-gmjp krb5kdc[27058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984573, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:49:33.462069 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:33.449656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:56309 (local address 127.1.183.190:34565)
0806 02:49:33.449825 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:33.449829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:33.450710 (+ 881us) server_negotiation.cc:407] Connection header received
0806 02:49:33.451694 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:33.451698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:33.451756 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:33.451868 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:33.453116 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:33.453647 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:33.454828 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:33.455054 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:33.455167 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:33.458057 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:33.458060 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:33.458062 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:33.458083 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:33.459775 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:33.460322 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:33.460329 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:33.460334 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:33.460442 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:33.460925 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:33.460928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:33.460929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:33.461108 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:33.461184 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:33.461839 (+ 655us) server_negotiation.cc:299] Negotiation successful
0806 02:49:33.461945 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:49:33.463256 27398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:33.450078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34565 (local address 127.1.183.129:56309)
0806 02:49:33.450558 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:33.450600 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:33.451451 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:33.452038 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:33.452045 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:33.452497 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:49:33.452958 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:33.452971 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:33.453777 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:33.453788 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:33.454682 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:33.454691 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:33.455226 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:33.455230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:33.455400 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:33.456244 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:49:33.456274 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:49:33.457880 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:49:33.459902 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:33.459907 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:33.459918 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:33.460211 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:33.460554 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:33.460561 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:33.460565 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:33.460792 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:33.461214 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:49:33.461221 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:49:33.461563 (+ 342us) client_negotiation.cc:769] Sending connection context
0806 02:49:33.461790 (+ 227us) client_negotiation.cc:240] Negotiation successful
0806 02:49:33.462157 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":130,"threads_started":1}
I20260806 02:49:33.464670 27396 heartbeater.cc:347] Connected to a master server at 127.1.183.190:34565
I20260806 02:49:33.464916 27396 heartbeater.cc:464] Registering TS with master...
I20260806 02:49:33.465432 27396 heartbeater.cc:511] Master 127.1.183.190:34565 requested a full tablet report, sending...
I20260806 02:49:33.467111 27127 ts_manager.cc:194] Registered new tserver with Master: 757473c25e6848909ab475b49bede65d (127.1.183.129:32781)
I20260806 02:49:33.468328 27127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.129@KRBTEST.COM'} at 127.1.183.129:56309
WARNING: no policy specified for kudu/127.1.183.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
I20260806 02:49:33.488741 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.130:0
--local_ip_for_outbound_sockets=127.1.183.130
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:34565
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:43967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:49:33.602175 27402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:33.602432 27402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:33.602474 27402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:33.606477 27402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:33.606530 27402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:33.606585 27402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.130
I20260806 02:49:33.611644 27402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:43967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:34565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.130
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:33.612656 27402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:33.613627 27402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:33.623616 27409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:33.623610 27416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:33.623610 27408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:33.623818 27402 server_base.cc:1034] running on GCE node
I20260806 02:49:33.624455 27402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:33.625138 27402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:33.626377 27402 hybrid_clock.cc:648] HybridClock initialized: now 1785984573626350 us; error 28 us; skew 500 ppm
Aug 06 02:49:33 dist-test-slave-gmjp krb5kdc[27058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984573, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:33.629339 27402 init.cc:377] Logged in from keytab as kudu/127.1.183.130@KRBTEST.COM (short username kudu)
I20260806 02:49:33.630277 27402 webserver.cc:460] Webserver started at http://127.1.183.130:46027/ using document root <none> and password file <none>
I20260806 02:49:33.630865 27402 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:33.630960 27402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:33.631202 27402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:33.633343 27402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-1/data/instance:
uuid: "7d3f857169b5457eb48c3f5b8e6aa96d"
format_stamp: "Formatted at 2026-08-06 02:49:33 on dist-test-slave-gmjp"
server_key: "72ef5ea928345667d732f02584353ef7"
server_key_iv: "5db112f47a14959b58b3a0fe5c257e83"
server_key_version: "encryptionkey@0"
I20260806 02:49:33.633934 27402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance:
uuid: "7d3f857169b5457eb48c3f5b8e6aa96d"
format_stamp: "Formatted at 2026-08-06 02:49:33 on dist-test-slave-gmjp"
server_key: "72ef5ea928345667d732f02584353ef7"
server_key_iv: "5db112f47a14959b58b3a0fe5c257e83"
server_key_version: "encryptionkey@0"
I20260806 02:49:33.637641 27402 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:49:33.640090 27427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:33.641337 27402 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:49:33.641516 27402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-1/wal
uuid: "7d3f857169b5457eb48c3f5b8e6aa96d"
format_stamp: "Formatted at 2026-08-06 02:49:33 on dist-test-slave-gmjp"
server_key: "72ef5ea928345667d732f02584353ef7"
server_key_iv: "5db112f47a14959b58b3a0fe5c257e83"
server_key_version: "encryptionkey@0"
I20260806 02:49:33.641635 27402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:33.658269 27402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:33.661180 27402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:33.661402 27402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:33.662014 27402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:49:33.662889 27402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:49:33.662961 27402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:33.663036 27402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:49:33.663079 27402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:33.672500 27402 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.130:36629
I20260806 02:49:33.672521 27591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.130:36629 every 8 connection(s)
I20260806 02:49:33.673427 27402 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
I20260806 02:49:33.676357 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 27402
I20260806 02:49:33.676554 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance
I20260806 02:49:33.676885 1758 external_mini_cluster.cc:1428] Setting key 58c57483021e7c4dfd18da0fae1f14dd
Aug 06 02:49:33 dist-test-slave-gmjp krb5kdc[27058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984573, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.183.131@KRBTEST.COM; defaulting to no policy
I20260806 02:49:33.689344 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:33.675293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:45759 (local address 127.1.183.190:34565)
0806 02:49:33.675455 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:33.675460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:33.676448 (+ 988us) server_negotiation.cc:407] Connection header received
0806 02:49:33.677540 (+ 1092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:33.677544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:33.677608 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:33.677701 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:33.679092 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:33.679921 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:33.681557 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:33.681780 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:33.681873 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:33.684997 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:33.685004 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:33.685010 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:33.685063 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:33.686822 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:33.687384 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:33.687390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:33.687392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:33.687484 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:33.687903 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:33.687907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:33.687909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:33.688187 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:33.688309 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:33.689067 (+ 758us) server_negotiation.cc:299] Negotiation successful
0806 02:49:33.689205 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:49:33.690219 27595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:33.675715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34565 (local address 127.1.183.130:45759)
0806 02:49:33.676274 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:33.676319 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:33.677289 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:33.677880 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:33.677889 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:33.678399 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:49:33.678939 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:33.678953 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:33.680058 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:33.680061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:33.681404 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:33.681420 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:33.681941 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:33.681947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:33.682193 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:33.682914 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:49:33.682940 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:49:33.684814 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:49:33.686956 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:33.686961 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:33.686974 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:33.687234 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:33.687588 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:33.687590 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:33.687592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:33.687788 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:33.688348 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:49:33.688355 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:49:33.688724 (+ 369us) client_negotiation.cc:769] Sending connection context
0806 02:49:33.688985 (+ 261us) client_negotiation.cc:240] Negotiation successful
0806 02:49:33.689254 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":121,"threads_started":1}
I20260806 02:49:33.691529 27592 heartbeater.cc:347] Connected to a master server at 127.1.183.190:34565
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.131@KRBTEST.COM" created.
I20260806 02:49:33.691795 27592 heartbeater.cc:464] Registering TS with master...
I20260806 02:49:33.692433 27592 heartbeater.cc:511] Master 127.1.183.190:34565 requested a full tablet report, sending...
I20260806 02:49:33.693637 27127 ts_manager.cc:194] Registered new tserver with Master: 7d3f857169b5457eb48c3f5b8e6aa96d (127.1.183.130:36629)
I20260806 02:49:33.694500 27127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.130@KRBTEST.COM'} at 127.1.183.130:45759
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
I20260806 02:49:33.707801 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.131:0
--local_ip_for_outbound_sockets=127.1.183.131
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:34565
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:43967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:49:33.822535 27598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:33.822778 27598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:33.822822 27598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:33.826850 27598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:33.826915 27598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:33.826972 27598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.131
I20260806 02:49:33.832425 27598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:43967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:34565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.131
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:33.833468 27598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:33.834450 27598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:33.844422 27607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:33.844498 27610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:33.844422 27614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:33.844446 27598 server_base.cc:1034] running on GCE node
I20260806 02:49:33.845039 27598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:33.845726 27598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:33.846923 27598 hybrid_clock.cc:648] HybridClock initialized: now 1785984573846899 us; error 43 us; skew 500 ppm
Aug 06 02:49:33 dist-test-slave-gmjp krb5kdc[27058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984573, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:33.849712 27598 init.cc:377] Logged in from keytab as kudu/127.1.183.131@KRBTEST.COM (short username kudu)
I20260806 02:49:33.850597 27598 webserver.cc:460] Webserver started at http://127.1.183.131:41187/ using document root <none> and password file <none>
I20260806 02:49:33.851202 27598 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:33.851295 27598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:33.851539 27598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:33.853719 27598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-2/data/instance:
uuid: "f3a997b56b4044b5855e288b66af968f"
format_stamp: "Formatted at 2026-08-06 02:49:33 on dist-test-slave-gmjp"
server_key: "3678789ce4d26ba5243ab32fdaaf5f16"
server_key_iv: "fb1fa40c65c47ba153327aa9ac6a010a"
server_key_version: "encryptionkey@0"
I20260806 02:49:33.854298 27598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance:
uuid: "f3a997b56b4044b5855e288b66af968f"
format_stamp: "Formatted at 2026-08-06 02:49:33 on dist-test-slave-gmjp"
server_key: "3678789ce4d26ba5243ab32fdaaf5f16"
server_key_iv: "fb1fa40c65c47ba153327aa9ac6a010a"
server_key_version: "encryptionkey@0"
I20260806 02:49:33.858083 27598 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:49:33.860509 27623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:33.861743 27598 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:49:33.861903 27598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-2/wal
uuid: "f3a997b56b4044b5855e288b66af968f"
format_stamp: "Formatted at 2026-08-06 02:49:33 on dist-test-slave-gmjp"
server_key: "3678789ce4d26ba5243ab32fdaaf5f16"
server_key_iv: "fb1fa40c65c47ba153327aa9ac6a010a"
server_key_version: "encryptionkey@0"
I20260806 02:49:33.862054 27598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:33.902562 27598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:33.905792 27598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:33.906010 27598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:33.906623 27598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:49:33.907480 27598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:49:33.907550 27598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:33.907622 27598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:49:33.907676 27598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:33.917078 27598 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.131:45309
I20260806 02:49:33.917104 27786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.131:45309 every 8 connection(s)
I20260806 02:49:33.917982 27598 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
I20260806 02:49:33.925700 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 27598
I20260806 02:49:33.925870 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance
I20260806 02:49:33.926234 1758 external_mini_cluster.cc:1428] Setting key 1c5252b6cef8418f0e109905f085753c
Aug 06 02:49:33 dist-test-slave-gmjp krb5kdc[27058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984573, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:49:33.931831 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:33.919888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:53185 (local address 127.1.183.190:34565)
0806 02:49:33.920027 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:33.920031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:33.920927 (+ 896us) server_negotiation.cc:407] Connection header received
0806 02:49:33.921964 (+ 1037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:33.921967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:33.922026 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:33.922130 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:33.923382 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:33.923911 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:33.925201 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:33.925431 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:33.925537 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:33.928215 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:33.928218 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:33.928221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:33.928246 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:33.929764 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:33.930296 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:33.930300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:33.930302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:33.930390 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:33.930734 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:33.930739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:33.930741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:33.930930 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:33.931038 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:33.931544 (+ 506us) server_negotiation.cc:299] Negotiation successful
0806 02:49:33.931667 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:49:33.932760 27791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:33.920317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34565 (local address 127.1.183.131:53185)
0806 02:49:33.920770 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:33.920813 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:33.921732 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:33.922352 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:33.922363 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:33.922744 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:49:33.923222 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:33.923235 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:33.924043 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:33.924052 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:33.925060 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:33.925070 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:33.925594 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:33.925598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:33.925762 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:33.926485 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:49:33.926502 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:49:33.928014 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:49:33.929898 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:33.929905 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:33.929921 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:33.930185 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:33.930485 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:33.930489 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:33.930491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:33.930636 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:33.931039 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:49:33.931047 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:49:33.931304 (+ 257us) client_negotiation.cc:769] Sending connection context
0806 02:49:33.931502 (+ 198us) client_negotiation.cc:240] Negotiation successful
0806 02:49:33.931844 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":101,"threads_started":1}
I20260806 02:49:33.933894 27787 heartbeater.cc:347] Connected to a master server at 127.1.183.190:34565
I20260806 02:49:33.934119 27787 heartbeater.cc:464] Registering TS with master...
I20260806 02:49:33.934588 27787 heartbeater.cc:511] Master 127.1.183.190:34565 requested a full tablet report, sending...
I20260806 02:49:33.935602 27127 ts_manager.cc:194] Registered new tserver with Master: f3a997b56b4044b5855e288b66af968f (127.1.183.131:45309)
I20260806 02:49:33.936131 27127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.131@KRBTEST.COM'} at 127.1.183.131:53185
I20260806 02:49:33.940704 1758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:49:33.949980 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:33.942703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58300 (local address 127.1.183.190:34565)
0806 02:49:33.942819 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:33.942823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:33.943044 (+ 221us) server_negotiation.cc:407] Connection header received
0806 02:49:33.943145 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:33.943147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:33.943186 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:33.943243 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:33.943992 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:33.944563 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:33.945525 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:33.945777 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:33.945896 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:33.947215 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:33.947219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:33.947222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:33.947247 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:33.948459 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:33.948924 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:33.948927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:33.948929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:33.949008 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:33.949245 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:33.949248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:33.949249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:33.949456 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:33.949558 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:33.949759 (+ 201us) server_negotiation.cc:299] Negotiation successful
0806 02:49:33.949855 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 02:49:33.952775 27127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58300:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:49:33.955830 27127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:49:33.969913 27805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:33.963771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:45309 (local address 127.0.0.1:52154)
0806 02:49:33.964555 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:33.964608 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:33.964819 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:33.965638 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:33.965645 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:33.965714 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:33.965953 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:33.965961 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:33.967270 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:33.967275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:33.968582 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:33.968592 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:33.969502 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:33.969504 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:33.969587 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:33.969617 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 02:49:33.969699 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 02:49:33.969786 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"mutex_wait_us":67,"thread_start_us":81,"threads_started":1}
I20260806 02:49:33.969993 27806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:33.963933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52154 (local address 127.1.183.131:45309)
0806 02:49:33.964305 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:33.964318 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:33.964696 (+ 378us) server_negotiation.cc:407] Connection header received
0806 02:49:33.965011 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:33.965017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:33.965238 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:33.965405 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:33.966087 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:33.967147 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:33.968706 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:33.969387 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:33.969496 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:33.969634 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:33.969742 (+ 108us) server_negotiation.cc:299] Negotiation successful
0806 02:49:33.969832 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":137,"threads_started":1}
I20260806 02:49:33.970631 27807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:33.963957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:32781 (local address 127.0.0.1:40032)
0806 02:49:33.965189 (+ 1232us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:33.965203 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:33.965318 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:33.965975 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:33.965978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:33.965993 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:33.966098 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:33.966104 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:33.967554 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:33.967558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:33.968997 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:33.969009 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:33.970230 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:33.970235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:33.970339 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:33.970366 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 02:49:33.970445 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 02:49:33.970509 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1173,"mutex_wait_us":126,"thread_start_us":101,"threads_started":1}
I20260806 02:49:33.971498 27808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:33.964505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:36629 (local address 127.0.0.1:54984)
0806 02:49:33.965442 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:33.965458 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:33.965536 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:33.966301 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:33.966304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:33.966327 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:33.966404 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:33.966410 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:33.967732 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:33.967735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:33.968997 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:33.969009 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:33.971116 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:33.971119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:33.971203 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:33.971225 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:49:33.971307 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 02:49:33.971383 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":879,"mutex_wait_us":30,"thread_start_us":92,"threads_started":1}
I20260806 02:49:33.971606 27810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:33.965062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54984 (local address 127.1.183.130:36629)
0806 02:49:33.965823 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:33.965834 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:33.965856 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:49:33.965943 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:33.965948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:33.966138 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:33.966318 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:33.966525 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:33.967590 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:33.970170 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:33.970999 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:33.971108 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:33.971246 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:33.971373 (+ 127us) server_negotiation.cc:299] Negotiation successful
0806 02:49:33.971466 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":135,"threads_started":1}
I20260806 02:49:33.971674 27809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:33.964630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40032 (local address 127.1.183.129:32781)
0806 02:49:33.965055 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:33.965064 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:33.965435 (+ 371us) server_negotiation.cc:407] Connection header received
0806 02:49:33.965501 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:33.965504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:33.965636 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:33.965796 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:33.966239 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:33.967423 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:33.969132 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:33.969895 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:33.970007 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:33.971413 (+ 1406us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:33.971458 (+ 45us) server_negotiation.cc:299] Negotiation successful
0806 02:49:33.971534 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":117,"threads_started":1}
I20260806 02:49:33.973230 27692 tablet_service.cc:1467] Processing CreateTablet for tablet 6d6fa0759b0148ae8918a008f439e0bc (DEFAULT_TABLE table=test-table [id=a2f32a8f852549bfbb290e0914ca9869]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:49:33.974215 27692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d6fa0759b0148ae8918a008f439e0bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:33.974308 27299 tablet_service.cc:1467] Processing CreateTablet for tablet 6d6fa0759b0148ae8918a008f439e0bc (DEFAULT_TABLE table=test-table [id=a2f32a8f852549bfbb290e0914ca9869]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:49:33.974287 27498 tablet_service.cc:1467] Processing CreateTablet for tablet 6d6fa0759b0148ae8918a008f439e0bc (DEFAULT_TABLE table=test-table [id=a2f32a8f852549bfbb290e0914ca9869]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:49:33.975308 27498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d6fa0759b0148ae8918a008f439e0bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:33.975308 27299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d6fa0759b0148ae8918a008f439e0bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:33.980239 27811 tablet_bootstrap.cc:492] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f: Bootstrap starting.
I20260806 02:49:33.982458 27813 tablet_bootstrap.cc:492] T 6d6fa0759b0148ae8918a008f439e0bc P 757473c25e6848909ab475b49bede65d: Bootstrap starting.
I20260806 02:49:33.982625 27811 tablet_bootstrap.cc:654] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:33.984043 27811 log.cc:824] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:33.984529 27814 tablet_bootstrap.cc:492] T 6d6fa0759b0148ae8918a008f439e0bc P 7d3f857169b5457eb48c3f5b8e6aa96d: Bootstrap starting.
I20260806 02:49:33.984927 27813 tablet_bootstrap.cc:654] T 6d6fa0759b0148ae8918a008f439e0bc P 757473c25e6848909ab475b49bede65d: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:33.985746 27813 log.cc:824] T 6d6fa0759b0148ae8918a008f439e0bc P 757473c25e6848909ab475b49bede65d: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:33.986315 27811 tablet_bootstrap.cc:492] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f: No bootstrap required, opened a new log
I20260806 02:49:33.986498 27814 tablet_bootstrap.cc:654] T 6d6fa0759b0148ae8918a008f439e0bc P 7d3f857169b5457eb48c3f5b8e6aa96d: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:33.986569 27811 ts_tablet_manager.cc:1397] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 02:49:33.987632 27814 log.cc:824] T 6d6fa0759b0148ae8918a008f439e0bc P 7d3f857169b5457eb48c3f5b8e6aa96d: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:33.987807 27813 tablet_bootstrap.cc:492] T 6d6fa0759b0148ae8918a008f439e0bc P 757473c25e6848909ab475b49bede65d: No bootstrap required, opened a new log
I20260806 02:49:33.988071 27813 ts_tablet_manager.cc:1397] T 6d6fa0759b0148ae8918a008f439e0bc P 757473c25e6848909ab475b49bede65d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:49:33.989635 27814 tablet_bootstrap.cc:492] T 6d6fa0759b0148ae8918a008f439e0bc P 7d3f857169b5457eb48c3f5b8e6aa96d: No bootstrap required, opened a new log
I20260806 02:49:33.989822 27814 ts_tablet_manager.cc:1397] T 6d6fa0759b0148ae8918a008f439e0bc P 7d3f857169b5457eb48c3f5b8e6aa96d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:49:33.990945 27811 raft_consensus.cc:348] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3a997b56b4044b5855e288b66af968f" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45309 } } peers { permanent_uuid: "757473c25e6848909ab475b49bede65d" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 32781 } } peers { permanent_uuid: "7d3f857169b5457eb48c3f5b8e6aa96d" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 36629 } }
I20260806 02:49:33.991230 27811 raft_consensus.cc:374] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:33.991315 27811 raft_consensus.cc:729] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3a997b56b4044b5855e288b66af968f, State: Initialized, Role: FOLLOWER
I20260806 02:49:33.991804 27811 consensus_queue.cc:260] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3a997b56b4044b5855e288b66af968f" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45309 } } peers { permanent_uuid: "757473c25e6848909ab475b49bede65d" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 32781 } } peers { permanent_uuid: "7d3f857169b5457eb48c3f5b8e6aa96d" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 36629 } }
I20260806 02:49:33.992525 27787 heartbeater.cc:503] Master 127.1.183.190:34565 was elected leader, sending a full tablet report...
I20260806 02:49:33.992897 27811 ts_tablet_manager.cc:1428] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 02:49:33.992755 27813 raft_consensus.cc:348] T 6d6fa0759b0148ae8918a008f439e0bc P 757473c25e6848909ab475b49bede65d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3a997b56b4044b5855e288b66af968f" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45309 } } peers { permanent_uuid: "757473c25e6848909ab475b49bede65d" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 32781 } } peers { permanent_uuid: "7d3f857169b5457eb48c3f5b8e6aa96d" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 36629 } }
I20260806 02:49:33.993023 27813 raft_consensus.cc:374] T 6d6fa0759b0148ae8918a008f439e0bc P 757473c25e6848909ab475b49bede65d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:33.993121 27813 raft_consensus.cc:729] T 6d6fa0759b0148ae8918a008f439e0bc P 757473c25e6848909ab475b49bede65d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 757473c25e6848909ab475b49bede65d, State: Initialized, Role: FOLLOWER
I20260806 02:49:33.993613 27813 consensus_queue.cc:260] T 6d6fa0759b0148ae8918a008f439e0bc P 757473c25e6848909ab475b49bede65d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3a997b56b4044b5855e288b66af968f" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45309 } } peers { permanent_uuid: "757473c25e6848909ab475b49bede65d" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 32781 } } peers { permanent_uuid: "7d3f857169b5457eb48c3f5b8e6aa96d" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 36629 } }
I20260806 02:49:33.993718 27814 raft_consensus.cc:348] T 6d6fa0759b0148ae8918a008f439e0bc P 7d3f857169b5457eb48c3f5b8e6aa96d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3a997b56b4044b5855e288b66af968f" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45309 } } peers { permanent_uuid: "757473c25e6848909ab475b49bede65d" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 32781 } } peers { permanent_uuid: "7d3f857169b5457eb48c3f5b8e6aa96d" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 36629 } }
I20260806 02:49:33.993991 27814 raft_consensus.cc:374] T 6d6fa0759b0148ae8918a008f439e0bc P 7d3f857169b5457eb48c3f5b8e6aa96d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:33.994103 27814 raft_consensus.cc:729] T 6d6fa0759b0148ae8918a008f439e0bc P 7d3f857169b5457eb48c3f5b8e6aa96d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d3f857169b5457eb48c3f5b8e6aa96d, State: Initialized, Role: FOLLOWER
I20260806 02:49:33.994396 27396 heartbeater.cc:503] Master 127.1.183.190:34565 was elected leader, sending a full tablet report...
I20260806 02:49:33.994554 27814 consensus_queue.cc:260] T 6d6fa0759b0148ae8918a008f439e0bc P 7d3f857169b5457eb48c3f5b8e6aa96d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3a997b56b4044b5855e288b66af968f" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45309 } } peers { permanent_uuid: "757473c25e6848909ab475b49bede65d" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 32781 } } peers { permanent_uuid: "7d3f857169b5457eb48c3f5b8e6aa96d" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 36629 } }
I20260806 02:49:33.994642 27813 ts_tablet_manager.cc:1428] T 6d6fa0759b0148ae8918a008f439e0bc P 757473c25e6848909ab475b49bede65d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:49:33.995158 27592 heartbeater.cc:503] Master 127.1.183.190:34565 was elected leader, sending a full tablet report...
I20260806 02:49:33.995465 27814 ts_tablet_manager.cc:1428] T 6d6fa0759b0148ae8918a008f439e0bc P 7d3f857169b5457eb48c3f5b8e6aa96d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260806 02:49:34.169008 27788 tablet.cc:2369] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:49:34.174671 27593 tablet.cc:2369] T 6d6fa0759b0148ae8918a008f439e0bc P 7d3f857169b5457eb48c3f5b8e6aa96d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:49:34.199226 27397 tablet.cc:2369] T 6d6fa0759b0148ae8918a008f439e0bc P 757473c25e6848909ab475b49bede65d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:49:34.277876 27818 raft_consensus.cc:482] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:49:34.278151 27818 raft_consensus.cc:504] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3a997b56b4044b5855e288b66af968f" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45309 } } peers { permanent_uuid: "757473c25e6848909ab475b49bede65d" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 32781 } } peers { permanent_uuid: "7d3f857169b5457eb48c3f5b8e6aa96d" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 36629 } }
I20260806 02:49:34.279381 27818 leader_election.cc:288] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 757473c25e6848909ab475b49bede65d (127.1.183.129:32781), 7d3f857169b5457eb48c3f5b8e6aa96d (127.1.183.130:36629)
I20260806 02:49:34.279577 27819 raft_consensus.cc:482] T 6d6fa0759b0148ae8918a008f439e0bc P 757473c25e6848909ab475b49bede65d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:49:34.279814 27819 raft_consensus.cc:504] T 6d6fa0759b0148ae8918a008f439e0bc P 757473c25e6848909ab475b49bede65d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3a997b56b4044b5855e288b66af968f" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45309 } } peers { permanent_uuid: "757473c25e6848909ab475b49bede65d" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 32781 } } peers { permanent_uuid: "7d3f857169b5457eb48c3f5b8e6aa96d" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 36629 } }
I20260806 02:49:34.281163 27819 leader_election.cc:288] T 6d6fa0759b0148ae8918a008f439e0bc P 757473c25e6848909ab475b49bede65d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3a997b56b4044b5855e288b66af968f (127.1.183.131:45309), 7d3f857169b5457eb48c3f5b8e6aa96d (127.1.183.130:36629)
I20260806 02:49:34.284298 27791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:34.279662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:36629 (local address 127.1.183.131:49181)
0806 02:49:34.279928 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:34.279945 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:34.280048 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:34.280412 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:34.280415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:34.280443 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:34.280567 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:34.280574 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:34.281588 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:34.281592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:34.282862 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:34.282873 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:34.283878 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:34.283883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:34.283989 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:34.284011 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:49:34.284080 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 02:49:34.284143 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181}
I20260806 02:49:34.284475 27809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:34.279751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:58607 (local address 127.1.183.129:32781)
0806 02:49:34.279877 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:34.279882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:34.280446 (+ 564us) server_negotiation.cc:407] Connection header received
0806 02:49:34.280641 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:34.280645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:34.280698 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:34.280773 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:34.281080 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:34.281753 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:34.283136 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:34.283800 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:34.283921 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:34.283966 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:34.284288 (+ 322us) server_negotiation.cc:299] Negotiation successful
0806 02:49:34.284359 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":46}
I20260806 02:49:34.284886 27823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:34.279662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:32781 (local address 127.1.183.131:58607)
0806 02:49:34.280350 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:34.280363 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:34.280521 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:34.280806 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:34.280808 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:34.280826 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:34.280949 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:34.280955 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:34.281882 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:34.281885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:34.283013 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:34.283021 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:34.284024 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:34.284027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:34.284132 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:34.284175 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 02:49:34.284646 (+ 471us) client_negotiation.cc:240] Negotiation successful
0806 02:49:34.284739 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":246,"thread_start_us":141,"threads_started":1}
I20260806 02:49:34.285036 27810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:34.279976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:49181 (local address 127.1.183.130:36629)
0806 02:49:34.280110 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:34.280116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:34.280131 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:49:34.280217 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:34.280220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:34.280275 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:34.280363 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:34.280714 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:34.281458 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:34.283002 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:34.283651 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:34.283768 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:34.283822 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:34.284836 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0806 02:49:34.284911 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:49:34.285637 27528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6fa0759b0148ae8918a008f439e0bc" candidate_uuid: "f3a997b56b4044b5855e288b66af968f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d3f857169b5457eb48c3f5b8e6aa96d" is_pre_election: true
I20260806 02:49:34.285632 27328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6fa0759b0148ae8918a008f439e0bc" candidate_uuid: "f3a997b56b4044b5855e288b66af968f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "757473c25e6848909ab475b49bede65d" is_pre_election: true
I20260806 02:49:34.285849 27328 raft_consensus.cc:2454] T 6d6fa0759b0148ae8918a008f439e0bc P 757473c25e6848909ab475b49bede65d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3a997b56b4044b5855e288b66af968f in term 0.
I20260806 02:49:34.285912 27528 raft_consensus.cc:2454] T 6d6fa0759b0148ae8918a008f439e0bc P 7d3f857169b5457eb48c3f5b8e6aa96d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3a997b56b4044b5855e288b66af968f in term 0.
I20260806 02:49:34.286089 27806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:34.281455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:57999 (local address 127.1.183.131:45309)
0806 02:49:34.281559 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:34.281565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:34.281826 (+ 261us) server_negotiation.cc:407] Connection header received
0806 02:49:34.282064 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:34.282068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:34.282118 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:34.282164 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:34.282602 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:34.283296 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:34.284741 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:34.285297 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:34.285394 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:34.285434 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:34.285920 (+ 486us) server_negotiation.cc:299] Negotiation successful
0806 02:49:34.285989 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 02:49:34.286334 27628 leader_election.cc:302] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 757473c25e6848909ab475b49bede65d, f3a997b56b4044b5855e288b66af968f; no voters:
I20260806 02:49:34.286576 27827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:34.281279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:45309 (local address 127.1.183.129:57999)
0806 02:49:34.281741 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:34.281754 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:34.281929 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:34.282310 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:34.282312 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:34.282329 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:34.282479 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:34.282486 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:34.283408 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:34.283412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:34.284609 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:34.284620 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:34.285646 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:34.285649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:34.285772 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:34.285805 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 02:49:34.286335 (+ 530us) client_negotiation.cc:240] Negotiation successful
0806 02:49:34.286419 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":26,"thread_start_us":120,"threads_started":1}
I20260806 02:49:34.286613 27818 raft_consensus.cc:2781] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:49:34.286688 27818 raft_consensus.cc:482] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:49:34.286736 27818 raft_consensus.cc:3037] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:34.287071 27829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:34.281662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:36629 (local address 127.1.183.129:39255)
0806 02:49:34.282330 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:34.282343 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:34.282432 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:34.282852 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:34.282855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:34.282873 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:34.282972 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:34.282977 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:34.284574 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:34.284578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:34.285755 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:34.285764 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:34.286662 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:34.286668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:34.286762 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:34.286787 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 02:49:34.286839 (+ 52us) client_negotiation.cc:240] Negotiation successful
0806 02:49:34.286915 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"mutex_wait_us":323,"thread_start_us":83,"threads_started":1}
I20260806 02:49:34.287322 27719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6fa0759b0148ae8918a008f439e0bc" candidate_uuid: "757473c25e6848909ab475b49bede65d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3a997b56b4044b5855e288b66af968f" is_pre_election: true
I20260806 02:49:34.287431 27828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:34.281813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:39255 (local address 127.1.183.130:36629)
0806 02:49:34.282632 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:34.282638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:34.282652 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:49:34.282689 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:34.282691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:34.282734 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:34.283069 (+ 335us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:34.283731 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:34.284431 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:34.285903 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:34.286515 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:34.287216 (+ 701us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:34.287262 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:34.287291 (+ 29us) server_negotiation.cc:299] Negotiation successful
0806 02:49:34.287338 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":523,"server-negotiator.queue_time_us":750,"thread_start_us":90,"threads_started":1}
I20260806 02:49:34.287904 27528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6fa0759b0148ae8918a008f439e0bc" candidate_uuid: "757473c25e6848909ab475b49bede65d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d3f857169b5457eb48c3f5b8e6aa96d" is_pre_election: true
I20260806 02:49:34.288043 27528 raft_consensus.cc:2454] T 6d6fa0759b0148ae8918a008f439e0bc P 7d3f857169b5457eb48c3f5b8e6aa96d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 757473c25e6848909ab475b49bede65d in term 0.
I20260806 02:49:34.288233 27818 raft_consensus.cc:504] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3a997b56b4044b5855e288b66af968f" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45309 } } peers { permanent_uuid: "757473c25e6848909ab475b49bede65d" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 32781 } } peers { permanent_uuid: "7d3f857169b5457eb48c3f5b8e6aa96d" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 36629 } }
I20260806 02:49:34.288514 27719 raft_consensus.cc:2379] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 757473c25e6848909ab475b49bede65d in current term 1: Already voted for candidate f3a997b56b4044b5855e288b66af968f in this term.
I20260806 02:49:34.288565 27231 leader_election.cc:302] T 6d6fa0759b0148ae8918a008f439e0bc P 757473c25e6848909ab475b49bede65d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 757473c25e6848909ab475b49bede65d, 7d3f857169b5457eb48c3f5b8e6aa96d; no voters:
I20260806 02:49:34.288651 27818 leader_election.cc:288] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f [CANDIDATE]: Term 1 election: Requested vote from peers 757473c25e6848909ab475b49bede65d (127.1.183.129:32781), 7d3f857169b5457eb48c3f5b8e6aa96d (127.1.183.130:36629)
I20260806 02:49:34.288933 27819 raft_consensus.cc:2781] T 6d6fa0759b0148ae8918a008f439e0bc P 757473c25e6848909ab475b49bede65d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:49:34.288998 27328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6fa0759b0148ae8918a008f439e0bc" candidate_uuid: "f3a997b56b4044b5855e288b66af968f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "757473c25e6848909ab475b49bede65d"
I20260806 02:49:34.289043 27819 raft_consensus.cc:482] T 6d6fa0759b0148ae8918a008f439e0bc P 757473c25e6848909ab475b49bede65d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:49:34.289089 27819 raft_consensus.cc:3037] T 6d6fa0759b0148ae8918a008f439e0bc P 757473c25e6848909ab475b49bede65d [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:34.289215 27528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6fa0759b0148ae8918a008f439e0bc" candidate_uuid: "f3a997b56b4044b5855e288b66af968f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d3f857169b5457eb48c3f5b8e6aa96d"
I20260806 02:49:34.289340 27528 raft_consensus.cc:3037] T 6d6fa0759b0148ae8918a008f439e0bc P 7d3f857169b5457eb48c3f5b8e6aa96d [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:34.290333 27819 raft_consensus.cc:504] T 6d6fa0759b0148ae8918a008f439e0bc P 757473c25e6848909ab475b49bede65d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3a997b56b4044b5855e288b66af968f" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45309 } } peers { permanent_uuid: "757473c25e6848909ab475b49bede65d" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 32781 } } peers { permanent_uuid: "7d3f857169b5457eb48c3f5b8e6aa96d" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 36629 } }
I20260806 02:49:34.290536 27528 raft_consensus.cc:2454] T 6d6fa0759b0148ae8918a008f439e0bc P 7d3f857169b5457eb48c3f5b8e6aa96d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3a997b56b4044b5855e288b66af968f in term 1.
I20260806 02:49:34.290549 27328 raft_consensus.cc:2379] T 6d6fa0759b0148ae8918a008f439e0bc P 757473c25e6848909ab475b49bede65d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f3a997b56b4044b5855e288b66af968f in current term 1: Already voted for candidate 757473c25e6848909ab475b49bede65d in this term.
I20260806 02:49:34.290683 27819 leader_election.cc:288] T 6d6fa0759b0148ae8918a008f439e0bc P 757473c25e6848909ab475b49bede65d [CANDIDATE]: Term 1 election: Requested vote from peers f3a997b56b4044b5855e288b66af968f (127.1.183.131:45309), 7d3f857169b5457eb48c3f5b8e6aa96d (127.1.183.130:36629)
I20260806 02:49:34.290884 27625 leader_election.cc:302] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d3f857169b5457eb48c3f5b8e6aa96d, f3a997b56b4044b5855e288b66af968f; no voters:
I20260806 02:49:34.291157 27719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6fa0759b0148ae8918a008f439e0bc" candidate_uuid: "757473c25e6848909ab475b49bede65d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3a997b56b4044b5855e288b66af968f"
I20260806 02:49:34.291167 27528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6fa0759b0148ae8918a008f439e0bc" candidate_uuid: "757473c25e6848909ab475b49bede65d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d3f857169b5457eb48c3f5b8e6aa96d"
I20260806 02:49:34.291318 27719 raft_consensus.cc:2379] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 757473c25e6848909ab475b49bede65d in current term 1: Already voted for candidate f3a997b56b4044b5855e288b66af968f in this term.
I20260806 02:49:34.291342 27528 raft_consensus.cc:2379] T 6d6fa0759b0148ae8918a008f439e0bc P 7d3f857169b5457eb48c3f5b8e6aa96d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 757473c25e6848909ab475b49bede65d in current term 1: Already voted for candidate f3a997b56b4044b5855e288b66af968f in this term.
I20260806 02:49:34.291505 27818 raft_consensus.cc:2781] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:49:34.291718 27818 raft_consensus.cc:686] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f [term 1 LEADER]: Becoming Leader. State: Replica: f3a997b56b4044b5855e288b66af968f, State: Running, Role: LEADER
I20260806 02:49:34.291785 27231 leader_election.cc:302] T 6d6fa0759b0148ae8918a008f439e0bc P 757473c25e6848909ab475b49bede65d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 757473c25e6848909ab475b49bede65d; no voters: 7d3f857169b5457eb48c3f5b8e6aa96d, f3a997b56b4044b5855e288b66af968f
I20260806 02:49:34.291940 27819 raft_consensus.cc:2726] T 6d6fa0759b0148ae8918a008f439e0bc P 757473c25e6848909ab475b49bede65d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 02:49:34.292068 27818 consensus_queue.cc:237] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3a997b56b4044b5855e288b66af968f" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45309 } } peers { permanent_uuid: "757473c25e6848909ab475b49bede65d" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 32781 } } peers { permanent_uuid: "7d3f857169b5457eb48c3f5b8e6aa96d" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 36629 } }
I20260806 02:49:34.295212 27127 catalog_manager.cc:5495] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f reported cstate change: term changed from 0 to 1, leader changed from <none> to f3a997b56b4044b5855e288b66af968f (127.1.183.131). New cstate: current_term: 1 leader_uuid: "f3a997b56b4044b5855e288b66af968f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3a997b56b4044b5855e288b66af968f" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 45309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "757473c25e6848909ab475b49bede65d" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 32781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d3f857169b5457eb48c3f5b8e6aa96d" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 36629 } health_report { overall_health: UNKNOWN } } }
I20260806 02:49:34.341827 27806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:34.337455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52158 (local address 127.1.183.131:45309)
0806 02:49:34.337593 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:34.337597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:34.337611 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:49:34.337723 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:34.337726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:34.337782 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:34.337871 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:49:34.338097 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:34.338669 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:34.339804 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:34.340244 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:34.340355 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:34.340793 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:49:34.341379 (+ 586us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:49:34.341519 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:34.341582 (+ 63us) server_negotiation.cc:299] Negotiation successful
0806 02:49:34.341641 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:49:34.348031 27328 raft_consensus.cc:1261] T 6d6fa0759b0148ae8918a008f439e0bc P 757473c25e6848909ab475b49bede65d [term 1 FOLLOWER]: Refusing update from remote peer f3a997b56b4044b5855e288b66af968f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:49:34.348031 27528 raft_consensus.cc:1261] T 6d6fa0759b0148ae8918a008f439e0bc P 7d3f857169b5457eb48c3f5b8e6aa96d [term 1 FOLLOWER]: Refusing update from remote peer f3a997b56b4044b5855e288b66af968f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:49:34.348663 27818 consensus_queue.cc:1035] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d3f857169b5457eb48c3f5b8e6aa96d" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 36629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:49:34.349114 27830 consensus_queue.cc:1035] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f [LEADER]: Connected to new peer: Peer: permanent_uuid: "757473c25e6848909ab475b49bede65d" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 32781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:49:34.360028 27836 mvcc.cc:205] Tried to move back new op lower bound from 7315392816522534912 to 7315392816303161344. Current Snapshot: MvccSnapshot[applied={T|T < 7315392816522534912}]
I20260806 02:49:34.367892 27127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:49:34.368069 27127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:49:34.370898 27127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 965476a1bbf349ed998400519b74356c: Sending DeleteTablet for 3 replicas of tablet 6d6fa0759b0148ae8918a008f439e0bc
I20260806 02:49:34.371840 27299 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d6fa0759b0148ae8918a008f439e0bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:49:34 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:40032
I20260806 02:49:34.371884 27498 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d6fa0759b0148ae8918a008f439e0bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:49:34 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:54984
I20260806 02:49:34.372574 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 27212
I20260806 02:49:34.372629 27692 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d6fa0759b0148ae8918a008f439e0bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:49:34 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:52154
I20260806 02:49:34.372691 27852 tablet_replica.cc:330] T 6d6fa0759b0148ae8918a008f439e0bc P 7d3f857169b5457eb48c3f5b8e6aa96d: stopping tablet replica
I20260806 02:49:34.375597 27853 tablet_replica.cc:330] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f: stopping tablet replica
I20260806 02:49:34.375733 27852 raft_consensus.cc:2229] T 6d6fa0759b0148ae8918a008f439e0bc P 7d3f857169b5457eb48c3f5b8e6aa96d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:49:34.375980 27853 raft_consensus.cc:2229] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:49:34.376179 27852 raft_consensus.cc:2258] T 6d6fa0759b0148ae8918a008f439e0bc P 7d3f857169b5457eb48c3f5b8e6aa96d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:49:34.376454 27853 raft_consensus.cc:2258] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:49:34.379689 27853 ts_tablet_manager.cc:1905] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:49:34.379714 27852 ts_tablet_manager.cc:1905] T 6d6fa0759b0148ae8918a008f439e0bc P 7d3f857169b5457eb48c3f5b8e6aa96d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:49:34.382611 27853 ts_tablet_manager.cc:1918] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:49:34.382793 27853 log.cc:1196] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-2/wal/wals/6d6fa0759b0148ae8918a008f439e0bc
I20260806 02:49:34.383203 27853 ts_tablet_manager.cc:1939] T 6d6fa0759b0148ae8918a008f439e0bc P f3a997b56b4044b5855e288b66af968f: Deleting consensus metadata
I20260806 02:49:34.383212 27852 ts_tablet_manager.cc:1918] T 6d6fa0759b0148ae8918a008f439e0bc P 7d3f857169b5457eb48c3f5b8e6aa96d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:49:34.383590 27852 log.cc:1196] T 6d6fa0759b0148ae8918a008f439e0bc P 7d3f857169b5457eb48c3f5b8e6aa96d: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984313887056-1758-0/minicluster-data/ts-1/wal/wals/6d6fa0759b0148ae8918a008f439e0bc
I20260806 02:49:34.384020 27852 ts_tablet_manager.cc:1939] T 6d6fa0759b0148ae8918a008f439e0bc P 7d3f857169b5457eb48c3f5b8e6aa96d: Deleting consensus metadata
I20260806 02:49:34.384608 27099 catalog_manager.cc:4841] TS f3a997b56b4044b5855e288b66af968f (127.1.183.131:45309): tablet 6d6fa0759b0148ae8918a008f439e0bc (table test-table [id=a2f32a8f852549bfbb290e0914ca9869]) successfully deleted
I20260806 02:49:34.385519 27099 catalog_manager.cc:4841] TS 7d3f857169b5457eb48c3f5b8e6aa96d (127.1.183.130:36629): tablet 6d6fa0759b0148ae8918a008f439e0bc (table test-table [id=a2f32a8f852549bfbb290e0914ca9869]) successfully deleted
W20260806 02:49:34.389319 27102 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.183.129:32781 (error 108)
I20260806 02:49:34.390376 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 27402
I20260806 02:49:34.390596 27808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:34.390210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:32781 (local address 127.0.0.1:40042)
0806 02:49:34.390380 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:34.390476 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.129:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260806 02:49:34.393848 27102 catalog_manager.cc:4584] TS 757473c25e6848909ab475b49bede65d (127.1.183.129:32781): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d6fa0759b0148ae8918a008f439e0bc: Network error: Client connection negotiation failed: client connection to 127.1.183.129:32781: connect: Connection refused (error 111)
I20260806 02:49:34.401839 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 27598
I20260806 02:49:34.418951 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 27078
2026-08-06T02:49:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2822 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-06T02:49:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:49:34Z Disabled control of system clock
I20260806 02:49:34.460812 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:38571
--webserver_interface=127.1.183.190
--webserver_port=0
--builtin_ntp_servers=127.1.183.148:38537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:38571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:49:34.573326 27867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:34.573588 27867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:34.573621 27867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:34.577471 27867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:49:34.577518 27867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:34.577533 27867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:34.577549 27867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:49:34.577564 27867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:49:34.582214 27867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:38537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:38571
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:38571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:34.583182 27867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:34.584499 27867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:34.592625 27876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:34.592607 27878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:34.592597 27874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:34.593024 27867 server_base.cc:1034] running on GCE node
I20260806 02:49:34.593631 27867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:34.594779 27867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:34.595963 27867 hybrid_clock.cc:648] HybridClock initialized: now 1785984574595937 us; error 45 us; skew 500 ppm
I20260806 02:49:34.597781 27867 webserver.cc:460] Webserver started at http://127.1.183.190:43591/ using document root <none> and password file <none>
I20260806 02:49:34.598361 27867 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:34.598467 27867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:34.598723 27867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:34.600795 27867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "244a89e1e7284b35b2e764b3ee75f18c"
format_stamp: "Formatted at 2026-08-06 02:49:34 on dist-test-slave-gmjp"
server_key: "7a3da35117b37ba8ac7b9e049a1d6f9d"
server_key_iv: "84b6b81554f862e19a3c7932f54c8e90"
server_key_version: "encryptionkey@0"
I20260806 02:49:34.601338 27867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "244a89e1e7284b35b2e764b3ee75f18c"
format_stamp: "Formatted at 2026-08-06 02:49:34 on dist-test-slave-gmjp"
server_key: "7a3da35117b37ba8ac7b9e049a1d6f9d"
server_key_iv: "84b6b81554f862e19a3c7932f54c8e90"
server_key_version: "encryptionkey@0"
I20260806 02:49:34.604947 27867 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:49:34.607407 27885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:34.608655 27867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:49:34.608805 27867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "244a89e1e7284b35b2e764b3ee75f18c"
format_stamp: "Formatted at 2026-08-06 02:49:34 on dist-test-slave-gmjp"
server_key: "7a3da35117b37ba8ac7b9e049a1d6f9d"
server_key_iv: "84b6b81554f862e19a3c7932f54c8e90"
server_key_version: "encryptionkey@0"
I20260806 02:49:34.608916 27867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:34.635190 27867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:34.635967 27867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:34.636245 27867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:34.643563 27867 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:38571
I20260806 02:49:34.643573 27962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:38571 every 8 connection(s)
I20260806 02:49:34.644559 27867 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:49:34.647070 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 27867
I20260806 02:49:34.647204 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:49:34.647478 1758 external_mini_cluster.cc:1428] Setting key 5017897b3d9951828651b42eb03745b7
I20260806 02:49:34.647780 27964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:34.654371 27964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 244a89e1e7284b35b2e764b3ee75f18c: Bootstrap starting.
I20260806 02:49:34.656741 27964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 244a89e1e7284b35b2e764b3ee75f18c: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:34.657145 27970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:34.649073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58030 (local address 127.1.183.190:38571)
0806 02:49:34.649728 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:34.649778 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:34.649815 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:49:34.650582 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:34.650617 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:34.651042 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:34.651384 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:34.651812 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:34.652876 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:34.653782 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:34.654258 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:34.654421 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:34.655355 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:34.655360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:34.655372 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:49:34.655759 (+ 387us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:34.655812 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:49:34.655832 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:34.655943 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:34.656060 (+ 117us) server_negotiation.cc:299] Negotiation successful
0806 02:49:34.656373 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":148,"threads_started":1}
I20260806 02:49:34.657618 27964 log.cc:824] T 00000000000000000000000000000000 P 244a89e1e7284b35b2e764b3ee75f18c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:34.659641 27964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 244a89e1e7284b35b2e764b3ee75f18c: No bootstrap required, opened a new log
I20260806 02:49:34.663262 27964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 244a89e1e7284b35b2e764b3ee75f18c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244a89e1e7284b35b2e764b3ee75f18c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38571 } }
I20260806 02:49:34.663475 27964 raft_consensus.cc:374] T 00000000000000000000000000000000 P 244a89e1e7284b35b2e764b3ee75f18c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:34.663553 27964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 244a89e1e7284b35b2e764b3ee75f18c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244a89e1e7284b35b2e764b3ee75f18c, State: Initialized, Role: FOLLOWER
I20260806 02:49:34.663959 27964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 244a89e1e7284b35b2e764b3ee75f18c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244a89e1e7284b35b2e764b3ee75f18c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38571 } }
I20260806 02:49:34.664103 27964 raft_consensus.cc:388] T 00000000000000000000000000000000 P 244a89e1e7284b35b2e764b3ee75f18c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:49:34.664191 27964 raft_consensus.cc:482] T 00000000000000000000000000000000 P 244a89e1e7284b35b2e764b3ee75f18c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:49:34.664328 27964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 244a89e1e7284b35b2e764b3ee75f18c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:34.665237 27964 raft_consensus.cc:504] T 00000000000000000000000000000000 P 244a89e1e7284b35b2e764b3ee75f18c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244a89e1e7284b35b2e764b3ee75f18c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38571 } }
I20260806 02:49:34.665568 27964 leader_election.cc:302] T 00000000000000000000000000000000 P 244a89e1e7284b35b2e764b3ee75f18c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 244a89e1e7284b35b2e764b3ee75f18c; no voters:
I20260806 02:49:34.665884 27964 leader_election.cc:288] T 00000000000000000000000000000000 P 244a89e1e7284b35b2e764b3ee75f18c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:49:34.666030 27975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 244a89e1e7284b35b2e764b3ee75f18c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:49:34.666258 27975 raft_consensus.cc:686] T 00000000000000000000000000000000 P 244a89e1e7284b35b2e764b3ee75f18c [term 1 LEADER]: Becoming Leader. State: Replica: 244a89e1e7284b35b2e764b3ee75f18c, State: Running, Role: LEADER
I20260806 02:49:34.666579 27975 consensus_queue.cc:237] T 00000000000000000000000000000000 P 244a89e1e7284b35b2e764b3ee75f18c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244a89e1e7284b35b2e764b3ee75f18c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38571 } }
I20260806 02:49:34.667212 27964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 244a89e1e7284b35b2e764b3ee75f18c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:49:34.668637 27979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 244a89e1e7284b35b2e764b3ee75f18c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 244a89e1e7284b35b2e764b3ee75f18c. Latest consensus state: current_term: 1 leader_uuid: "244a89e1e7284b35b2e764b3ee75f18c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244a89e1e7284b35b2e764b3ee75f18c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38571 } } }
I20260806 02:49:34.668790 27979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 244a89e1e7284b35b2e764b3ee75f18c [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:34.669709 27978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 244a89e1e7284b35b2e764b3ee75f18c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "244a89e1e7284b35b2e764b3ee75f18c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244a89e1e7284b35b2e764b3ee75f18c" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38571 } } }
I20260806 02:49:34.669817 27978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 244a89e1e7284b35b2e764b3ee75f18c [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:34.669827 27989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:49:34.672495 27989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:49:34.677505 27989 catalog_manager.cc:1347] Generated new cluster ID: 926bb58e91144c4799f241ff87121bfa
I20260806 02:49:34.677577 27989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:49:34.690361 27989 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:49:34.691612 27989 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:49:34.700387 27989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 244a89e1e7284b35b2e764b3ee75f18c: Generated new TSK 0
I20260806 02:49:34.701221 27989 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:49:34.706477 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.129:0
--local_ip_for_outbound_sockets=127.1.183.129
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:38571
--builtin_ntp_servers=127.1.183.148:38537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:49:34.821301 28007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:34.821585 28007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:34.821624 28007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:34.825724 28007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:34.825779 28007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:34.825870 28007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.129
I20260806 02:49:34.830593 28007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:38537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:38571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.129
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:34.831723 28007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:34.833242 28007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:34.843016 28015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:34.843016 28014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:34.843031 28017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:34.843251 28007 server_base.cc:1034] running on GCE node
I20260806 02:49:34.843741 28007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:34.844506 28007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:34.845667 28007 hybrid_clock.cc:648] HybridClock initialized: now 1785984574845653 us; error 25 us; skew 500 ppm
I20260806 02:49:34.847491 28007 webserver.cc:460] Webserver started at http://127.1.183.129:36581/ using document root <none> and password file <none>
I20260806 02:49:34.848116 28007 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:34.848258 28007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:34.848510 28007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:34.850594 28007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-0/data/instance:
uuid: "4e249f6ef94d4ccc95425ce495376756"
format_stamp: "Formatted at 2026-08-06 02:49:34 on dist-test-slave-gmjp"
server_key: "d8aac6d66d69613631ddf9d9c5d7a936"
server_key_iv: "8666a7f577c7c07cbe15d8306cd4bd26"
server_key_version: "encryptionkey@0"
I20260806 02:49:34.851189 28007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance:
uuid: "4e249f6ef94d4ccc95425ce495376756"
format_stamp: "Formatted at 2026-08-06 02:49:34 on dist-test-slave-gmjp"
server_key: "d8aac6d66d69613631ddf9d9c5d7a936"
server_key_iv: "8666a7f577c7c07cbe15d8306cd4bd26"
server_key_version: "encryptionkey@0"
I20260806 02:49:34.855084 28007 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 02:49:34.857532 28027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:34.858739 28007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:49:34.858860 28007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-0/wal
uuid: "4e249f6ef94d4ccc95425ce495376756"
format_stamp: "Formatted at 2026-08-06 02:49:34 on dist-test-slave-gmjp"
server_key: "d8aac6d66d69613631ddf9d9c5d7a936"
server_key_iv: "8666a7f577c7c07cbe15d8306cd4bd26"
server_key_version: "encryptionkey@0"
I20260806 02:49:34.858978 28007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:34.876394 28007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:34.877142 28007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:34.877349 28007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:34.877976 28007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:49:34.878965 28007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:49:34.879011 28007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:34.879083 28007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:49:34.879114 28007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:34.888470 28007 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.129:35563
I20260806 02:49:34.888481 28195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.129:35563 every 8 connection(s)
I20260806 02:49:34.889402 28007 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
I20260806 02:49:34.894171 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 28007
I20260806 02:49:34.894299 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance
I20260806 02:49:34.894624 1758 external_mini_cluster.cc:1428] Setting key f280ecfc47434b1c1bf7d3f3effd831c
I20260806 02:49:34.896742 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.130:0
--local_ip_for_outbound_sockets=127.1.183.130
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:38571
--builtin_ntp_servers=127.1.183.148:38537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:49:34.899712 27970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:34.891287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:59089 (local address 127.1.183.190:38571)
0806 02:49:34.891437 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:34.891441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:34.892534 (+ 1093us) server_negotiation.cc:407] Connection header received
0806 02:49:34.893964 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:34.893966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:34.894012 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:34.894088 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:34.895257 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:34.895800 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:34.897519 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:34.897738 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:34.897817 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:34.899117 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:34.899123 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:34.899128 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:49:34.899147 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:34.899174 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:49:34.899186 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:34.899275 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:34.899424 (+ 149us) server_negotiation.cc:299] Negotiation successful
0806 02:49:34.899520 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:49:34.900640 28201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:34.891706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:38571 (local address 127.1.183.129:59089)
0806 02:49:34.892408 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:34.892449 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:34.893720 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:34.894305 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:34.894316 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:34.894423 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:49:34.895105 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:34.895122 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:34.895938 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:34.895941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:34.897372 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:34.897388 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:34.897904 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:34.897911 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:34.898145 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:34.898846 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:49:34.898848 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:49:34.898892 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:49:34.898957 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:49:34.899282 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:49:34.899290 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:49:34.899294 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 02:49:34.899397 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 02:49:34.899618 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":105,"threads_started":1}
I20260806 02:49:34.901959 28197 heartbeater.cc:347] Connected to a master server at 127.1.183.190:38571
I20260806 02:49:34.902225 28197 heartbeater.cc:464] Registering TS with master...
I20260806 02:49:34.902758 28197 heartbeater.cc:511] Master 127.1.183.190:38571 requested a full tablet report, sending...
I20260806 02:49:34.904393 27908 ts_manager.cc:194] Registered new tserver with Master: 4e249f6ef94d4ccc95425ce495376756 (127.1.183.129:35563)
I20260806 02:49:34.905534 27908 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.183.129:59089
W20260806 02:49:35.012598 28202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:35.012938 28202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:35.013043 28202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:35.017092 28202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:35.017185 28202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:35.017285 28202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.130
I20260806 02:49:35.022043 28202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:38537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:38571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.130
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:35.023197 28202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:35.024658 28202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:35.034464 28212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:35.034461 28210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:35.034539 28209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:35.034729 28202 server_base.cc:1034] running on GCE node
I20260806 02:49:35.035185 28202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:35.035818 28202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:35.037000 28202 hybrid_clock.cc:648] HybridClock initialized: now 1785984575036972 us; error 37 us; skew 500 ppm
I20260806 02:49:35.038967 28202 webserver.cc:460] Webserver started at http://127.1.183.130:40423/ using document root <none> and password file <none>
I20260806 02:49:35.039588 28202 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:35.039654 28202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:35.039887 28202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:35.042007 28202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-1/data/instance:
uuid: "3816f29f4663442b96e575d0c48d9dd3"
format_stamp: "Formatted at 2026-08-06 02:49:35 on dist-test-slave-gmjp"
server_key: "7ec6db36dee95f35e3c87078778dc1e3"
server_key_iv: "b6c4d54bc7d8562ef18d90cb483725e7"
server_key_version: "encryptionkey@0"
I20260806 02:49:35.042563 28202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance:
uuid: "3816f29f4663442b96e575d0c48d9dd3"
format_stamp: "Formatted at 2026-08-06 02:49:35 on dist-test-slave-gmjp"
server_key: "7ec6db36dee95f35e3c87078778dc1e3"
server_key_iv: "b6c4d54bc7d8562ef18d90cb483725e7"
server_key_version: "encryptionkey@0"
I20260806 02:49:35.046253 28202 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 02:49:35.048668 28218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:35.049862 28202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:49:35.049985 28202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-1/wal
uuid: "3816f29f4663442b96e575d0c48d9dd3"
format_stamp: "Formatted at 2026-08-06 02:49:35 on dist-test-slave-gmjp"
server_key: "7ec6db36dee95f35e3c87078778dc1e3"
server_key_iv: "b6c4d54bc7d8562ef18d90cb483725e7"
server_key_version: "encryptionkey@0"
I20260806 02:49:35.050089 28202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:35.065828 28202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:35.066584 28202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:35.066792 28202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:35.067544 28202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:49:35.068640 28202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:49:35.068691 28202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:35.068730 28202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:49:35.068746 28202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:35.078095 28202 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.130:35871
I20260806 02:49:35.078126 28391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.130:35871 every 8 connection(s)
I20260806 02:49:35.079067 28202 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
I20260806 02:49:35.085407 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 28202
I20260806 02:49:35.085513 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance
I20260806 02:49:35.085795 1758 external_mini_cluster.cc:1428] Setting key 54ecf11cf4c3751fc9e25a525da7ebc9
I20260806 02:49:35.087864 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.131:0
--local_ip_for_outbound_sockets=127.1.183.131
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:38571
--builtin_ntp_servers=127.1.183.148:38537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:49:35.088651 27970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:35.080914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:37045 (local address 127.1.183.190:38571)
0806 02:49:35.081057 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:35.081060 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:35.082056 (+ 996us) server_negotiation.cc:407] Connection header received
0806 02:49:35.083478 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:35.083482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:35.083553 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:35.083645 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:35.084575 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:35.085144 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:35.086450 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:35.086725 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:35.086807 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:35.087993 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:35.087999 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:35.088001 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:49:35.088017 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:35.088045 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:49:35.088059 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:35.088134 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:35.088359 (+ 225us) server_negotiation.cc:299] Negotiation successful
0806 02:49:35.088431 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:49:35.089448 28394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:35.081250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:38571 (local address 127.1.183.130:37045)
0806 02:49:35.081899 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:35.081937 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:35.083242 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:35.083830 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:35.083837 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:35.083919 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:49:35.084417 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:35.084431 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:35.085305 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:35.085308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:35.086324 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:35.086335 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:35.086883 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:35.086888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:35.087090 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:35.087727 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:49:35.087728 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:49:35.087769 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:49:35.087833 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:49:35.088158 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:49:35.088200 (+ 42us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:49:35.088206 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 02:49:35.088325 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 02:49:35.088557 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":114,"threads_started":1}
I20260806 02:49:35.090728 28392 heartbeater.cc:347] Connected to a master server at 127.1.183.190:38571
I20260806 02:49:35.090982 28392 heartbeater.cc:464] Registering TS with master...
I20260806 02:49:35.091536 28392 heartbeater.cc:511] Master 127.1.183.190:38571 requested a full tablet report, sending...
I20260806 02:49:35.092720 27908 ts_manager.cc:194] Registered new tserver with Master: 3816f29f4663442b96e575d0c48d9dd3 (127.1.183.130:35871)
I20260806 02:49:35.093317 27908 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.183.130:37045
W20260806 02:49:35.208055 28396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:35.208374 28396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:35.208437 28396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:35.212487 28396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:35.212543 28396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:35.212657 28396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.131
I20260806 02:49:35.217545 28396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:38537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:38571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.131
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:35.218670 28396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:35.220042 28396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:35.229537 28408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:35.229619 28409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:35.229759 28396 server_base.cc:1034] running on GCE node
W20260806 02:49:35.229619 28413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:35.230333 28396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:35.231176 28396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:35.232360 28396 hybrid_clock.cc:648] HybridClock initialized: now 1785984575232334 us; error 41 us; skew 500 ppm
I20260806 02:49:35.234371 28396 webserver.cc:460] Webserver started at http://127.1.183.131:43659/ using document root <none> and password file <none>
I20260806 02:49:35.234999 28396 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:35.235069 28396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:35.235354 28396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:35.237674 28396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-2/data/instance:
uuid: "d9a180d89c524c99919ea0e829d846e4"
format_stamp: "Formatted at 2026-08-06 02:49:35 on dist-test-slave-gmjp"
server_key: "38f24732b3f91ce9a01ff507e5d8c891"
server_key_iv: "0aeadd85fa418a434b040451cbf6251b"
server_key_version: "encryptionkey@0"
I20260806 02:49:35.238303 28396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance:
uuid: "d9a180d89c524c99919ea0e829d846e4"
format_stamp: "Formatted at 2026-08-06 02:49:35 on dist-test-slave-gmjp"
server_key: "38f24732b3f91ce9a01ff507e5d8c891"
server_key_iv: "0aeadd85fa418a434b040451cbf6251b"
server_key_version: "encryptionkey@0"
I20260806 02:49:35.242448 28396 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:49:35.245090 28423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:35.246284 28396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:49:35.246421 28396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-2/wal
uuid: "d9a180d89c524c99919ea0e829d846e4"
format_stamp: "Formatted at 2026-08-06 02:49:35 on dist-test-slave-gmjp"
server_key: "38f24732b3f91ce9a01ff507e5d8c891"
server_key_iv: "0aeadd85fa418a434b040451cbf6251b"
server_key_version: "encryptionkey@0"
I20260806 02:49:35.246532 28396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:35.270917 28396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:35.271629 28396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:35.271787 28396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:35.272433 28396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:49:35.273380 28396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:49:35.273427 28396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:35.273473 28396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:49:35.273496 28396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:35.282673 28396 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.131:35261
I20260806 02:49:35.282697 28596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.131:35261 every 8 connection(s)
I20260806 02:49:35.283620 28396 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
I20260806 02:49:35.287184 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 28396
I20260806 02:49:35.287297 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance
I20260806 02:49:35.287607 1758 external_mini_cluster.cc:1428] Setting key 12d86d1899d336c38a35df2dcff2e2bb
I20260806 02:49:35.293617 27970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:35.285461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:48309 (local address 127.1.183.190:38571)
0806 02:49:35.285637 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:35.285641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:35.286676 (+ 1035us) server_negotiation.cc:407] Connection header received
0806 02:49:35.288518 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:35.288521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:35.288581 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:35.288684 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:35.289589 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:35.290212 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:35.291379 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:35.291626 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:35.291745 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:35.293081 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:35.293084 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:35.293086 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:49:35.293099 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:35.293118 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:49:35.293127 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:35.293211 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:35.293403 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 02:49:35.293472 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 02:49:35.294543 28599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:35.285869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:38571 (local address 127.1.183.131:48309)
0806 02:49:35.286530 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:35.286570 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:35.288280 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:35.288876 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:35.288883 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:35.288969 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:49:35.289451 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:35.289463 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:35.290349 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:35.290353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:35.291243 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:35.291252 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:35.291790 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:35.291794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:35.291972 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:35.292620 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:49:35.292622 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:49:35.292665 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:49:35.292740 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:49:35.293215 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:49:35.293227 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:49:35.293233 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 02:49:35.293401 (+ 168us) client_negotiation.cc:240] Negotiation successful
0806 02:49:35.293707 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":117,"threads_started":1}
I20260806 02:49:35.295758 28597 heartbeater.cc:347] Connected to a master server at 127.1.183.190:38571
I20260806 02:49:35.296043 28597 heartbeater.cc:464] Registering TS with master...
I20260806 02:49:35.296661 28597 heartbeater.cc:511] Master 127.1.183.190:38571 requested a full tablet report, sending...
I20260806 02:49:35.297803 27908 ts_manager.cc:194] Registered new tserver with Master: d9a180d89c524c99919ea0e829d846e4 (127.1.183.131:35261)
I20260806 02:49:35.298472 27908 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.183.131:48309
I20260806 02:49:35.301563 1758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:49:35.307080 27970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:35.303599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58046 (local address 127.1.183.190:38571)
0806 02:49:35.303735 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:35.303739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:35.303953 (+ 214us) server_negotiation.cc:407] Connection header received
0806 02:49:35.304048 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:35.304051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:35.304090 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:35.304211 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:35.304483 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:35.304969 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:35.305747 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:35.305949 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:35.306023 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:35.306748 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:35.306751 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:35.306753 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:49:35.306764 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:35.306782 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:49:35.306790 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:35.306851 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:35.306925 (+ 74us) server_negotiation.cc:299] Negotiation successful
0806 02:49:35.306970 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 02:49:35.310040 27908 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:49:35.313117 27908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:49:35.328510 28612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:35.322781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:35563 (local address 127.0.0.1:36146)
0806 02:49:35.323446 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:35.323459 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:35.323547 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:35.324687 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:35.324690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:35.324708 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:35.324797 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:35.324804 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:35.326023 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:35.326026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:35.326983 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:35.326994 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:35.327895 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:35.327899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:35.328009 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:35.328050 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 02:49:35.328203 (+ 153us) client_negotiation.cc:240] Negotiation successful
0806 02:49:35.328330 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":98,"thread_start_us":131,"threads_started":1}
I20260806 02:49:35.329223 28614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:35.321966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:35871 (local address 127.0.0.1:60158)
0806 02:49:35.323050 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:35.323106 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:35.323257 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:35.324028 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:35.324034 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:35.324071 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:35.324299 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:35.324306 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:35.325791 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:35.325795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:35.327471 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:35.327477 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:35.328798 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:35.328803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:35.328890 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:35.328911 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 02:49:35.328993 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 02:49:35.329075 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"mutex_wait_us":283,"spinlock_wait_cycles":6144,"thread_start_us":98,"threads_started":1}
I20260806 02:49:35.329286 28618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:35.323250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36146 (local address 127.1.183.129:35563)
0806 02:49:35.324281 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:35.324293 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:35.324317 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:49:35.324405 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:35.324410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:35.324555 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:35.324710 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:35.324943 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:35.325897 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:35.327112 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:35.327753 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:35.327880 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:35.328072 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:35.328229 (+ 157us) server_negotiation.cc:299] Negotiation successful
0806 02:49:35.328335 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":931,"thread_start_us":128,"threads_started":1}
I20260806 02:49:35.329347 28615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:35.322089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60158 (local address 127.1.183.130:35871)
0806 02:49:35.322855 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:35.322880 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:35.323215 (+ 335us) server_negotiation.cc:407] Connection header received
0806 02:49:35.323463 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:35.323469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:35.323679 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:35.323810 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:35.324466 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:35.325676 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:35.327616 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:35.328445 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:35.328554 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:35.328740 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:35.329056 (+ 316us) server_negotiation.cc:299] Negotiation successful
0806 02:49:35.329159 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":129,"threads_started":1}
I20260806 02:49:35.329938 28617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:35.322049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:35261 (local address 127.0.0.1:45052)
0806 02:49:35.323050 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:35.323105 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:35.323340 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:35.324028 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:35.324034 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:35.324070 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:35.324278 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:35.324287 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:35.325792 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:35.325795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:35.327463 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:35.327475 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:35.329084 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:35.329087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:35.329183 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:35.329205 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:49:35.329733 (+ 528us) client_negotiation.cc:240] Negotiation successful
0806 02:49:35.329810 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":932,"mutex_wait_us":187,"spinlock_wait_cycles":5248,"thread_start_us":95,"threads_started":1}
I20260806 02:49:35.330051 28616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:35.322298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45052 (local address 127.1.183.131:35261)
0806 02:49:35.322768 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:35.322783 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:35.323274 (+ 491us) server_negotiation.cc:407] Connection header received
0806 02:49:35.323484 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:35.323488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:35.323678 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:35.323810 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:35.324429 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:35.325647 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:35.327625 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:35.328416 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:35.328520 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:35.329017 (+ 497us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:35.329796 (+ 779us) server_negotiation.cc:299] Negotiation successful
0806 02:49:35.329900 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":133,"threads_started":1}
I20260806 02:49:35.332494 28494 tablet_service.cc:1467] Processing CreateTablet for tablet 12d362103f9647aca18a71ff40543991 (DEFAULT_TABLE table=test-table [id=79fe25b940e44ca89c767e739a39ab60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:49:35.332502 28092 tablet_service.cc:1467] Processing CreateTablet for tablet 12d362103f9647aca18a71ff40543991 (DEFAULT_TABLE table=test-table [id=79fe25b940e44ca89c767e739a39ab60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:49:35.332502 28302 tablet_service.cc:1467] Processing CreateTablet for tablet 12d362103f9647aca18a71ff40543991 (DEFAULT_TABLE table=test-table [id=79fe25b940e44ca89c767e739a39ab60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:49:35.333490 28494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12d362103f9647aca18a71ff40543991. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:35.333515 28092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12d362103f9647aca18a71ff40543991. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:35.333515 28302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12d362103f9647aca18a71ff40543991. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:35.339699 28619 tablet_bootstrap.cc:492] T 12d362103f9647aca18a71ff40543991 P d9a180d89c524c99919ea0e829d846e4: Bootstrap starting.
I20260806 02:49:35.341814 28622 tablet_bootstrap.cc:492] T 12d362103f9647aca18a71ff40543991 P 4e249f6ef94d4ccc95425ce495376756: Bootstrap starting.
I20260806 02:49:35.342206 28619 tablet_bootstrap.cc:654] T 12d362103f9647aca18a71ff40543991 P d9a180d89c524c99919ea0e829d846e4: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:35.342183 28621 tablet_bootstrap.cc:492] T 12d362103f9647aca18a71ff40543991 P 3816f29f4663442b96e575d0c48d9dd3: Bootstrap starting.
I20260806 02:49:35.343238 28619 log.cc:824] T 12d362103f9647aca18a71ff40543991 P d9a180d89c524c99919ea0e829d846e4: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:35.344110 28622 tablet_bootstrap.cc:654] T 12d362103f9647aca18a71ff40543991 P 4e249f6ef94d4ccc95425ce495376756: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:35.344445 28621 tablet_bootstrap.cc:654] T 12d362103f9647aca18a71ff40543991 P 3816f29f4663442b96e575d0c48d9dd3: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:35.345124 28622 log.cc:824] T 12d362103f9647aca18a71ff40543991 P 4e249f6ef94d4ccc95425ce495376756: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:35.345254 28621 log.cc:824] T 12d362103f9647aca18a71ff40543991 P 3816f29f4663442b96e575d0c48d9dd3: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:35.346642 28619 tablet_bootstrap.cc:492] T 12d362103f9647aca18a71ff40543991 P d9a180d89c524c99919ea0e829d846e4: No bootstrap required, opened a new log
I20260806 02:49:35.346885 28619 ts_tablet_manager.cc:1397] T 12d362103f9647aca18a71ff40543991 P d9a180d89c524c99919ea0e829d846e4: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 02:49:35.347283 28622 tablet_bootstrap.cc:492] T 12d362103f9647aca18a71ff40543991 P 4e249f6ef94d4ccc95425ce495376756: No bootstrap required, opened a new log
I20260806 02:49:35.347283 28621 tablet_bootstrap.cc:492] T 12d362103f9647aca18a71ff40543991 P 3816f29f4663442b96e575d0c48d9dd3: No bootstrap required, opened a new log
I20260806 02:49:35.347471 28622 ts_tablet_manager.cc:1397] T 12d362103f9647aca18a71ff40543991 P 4e249f6ef94d4ccc95425ce495376756: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260806 02:49:35.347471 28621 ts_tablet_manager.cc:1397] T 12d362103f9647aca18a71ff40543991 P 3816f29f4663442b96e575d0c48d9dd3: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260806 02:49:35.350530 28619 raft_consensus.cc:348] T 12d362103f9647aca18a71ff40543991 P d9a180d89c524c99919ea0e829d846e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3816f29f4663442b96e575d0c48d9dd3" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 35871 } } peers { permanent_uuid: "d9a180d89c524c99919ea0e829d846e4" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 35261 } } peers { permanent_uuid: "4e249f6ef94d4ccc95425ce495376756" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 35563 } }
I20260806 02:49:35.350812 28619 raft_consensus.cc:374] T 12d362103f9647aca18a71ff40543991 P d9a180d89c524c99919ea0e829d846e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:35.350868 28619 raft_consensus.cc:729] T 12d362103f9647aca18a71ff40543991 P d9a180d89c524c99919ea0e829d846e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9a180d89c524c99919ea0e829d846e4, State: Initialized, Role: FOLLOWER
I20260806 02:49:35.351382 28619 consensus_queue.cc:260] T 12d362103f9647aca18a71ff40543991 P d9a180d89c524c99919ea0e829d846e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3816f29f4663442b96e575d0c48d9dd3" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 35871 } } peers { permanent_uuid: "d9a180d89c524c99919ea0e829d846e4" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 35261 } } peers { permanent_uuid: "4e249f6ef94d4ccc95425ce495376756" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 35563 } }
I20260806 02:49:35.352037 28622 raft_consensus.cc:348] T 12d362103f9647aca18a71ff40543991 P 4e249f6ef94d4ccc95425ce495376756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3816f29f4663442b96e575d0c48d9dd3" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 35871 } } peers { permanent_uuid: "d9a180d89c524c99919ea0e829d846e4" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 35261 } } peers { permanent_uuid: "4e249f6ef94d4ccc95425ce495376756" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 35563 } }
I20260806 02:49:35.352037 28621 raft_consensus.cc:348] T 12d362103f9647aca18a71ff40543991 P 3816f29f4663442b96e575d0c48d9dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3816f29f4663442b96e575d0c48d9dd3" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 35871 } } peers { permanent_uuid: "d9a180d89c524c99919ea0e829d846e4" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 35261 } } peers { permanent_uuid: "4e249f6ef94d4ccc95425ce495376756" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 35563 } }
I20260806 02:49:35.352339 28621 raft_consensus.cc:374] T 12d362103f9647aca18a71ff40543991 P 3816f29f4663442b96e575d0c48d9dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:35.352339 28622 raft_consensus.cc:374] T 12d362103f9647aca18a71ff40543991 P 4e249f6ef94d4ccc95425ce495376756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:35.352407 28622 raft_consensus.cc:729] T 12d362103f9647aca18a71ff40543991 P 4e249f6ef94d4ccc95425ce495376756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e249f6ef94d4ccc95425ce495376756, State: Initialized, Role: FOLLOWER
I20260806 02:49:35.352407 28621 raft_consensus.cc:729] T 12d362103f9647aca18a71ff40543991 P 3816f29f4663442b96e575d0c48d9dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3816f29f4663442b96e575d0c48d9dd3, State: Initialized, Role: FOLLOWER
I20260806 02:49:35.352430 28619 ts_tablet_manager.cc:1428] T 12d362103f9647aca18a71ff40543991 P d9a180d89c524c99919ea0e829d846e4: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260806 02:49:35.352883 28597 heartbeater.cc:503] Master 127.1.183.190:38571 was elected leader, sending a full tablet report...
I20260806 02:49:35.352897 28621 consensus_queue.cc:260] T 12d362103f9647aca18a71ff40543991 P 3816f29f4663442b96e575d0c48d9dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3816f29f4663442b96e575d0c48d9dd3" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 35871 } } peers { permanent_uuid: "d9a180d89c524c99919ea0e829d846e4" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 35261 } } peers { permanent_uuid: "4e249f6ef94d4ccc95425ce495376756" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 35563 } }
I20260806 02:49:35.353576 28392 heartbeater.cc:503] Master 127.1.183.190:38571 was elected leader, sending a full tablet report...
I20260806 02:49:35.353698 28622 consensus_queue.cc:260] T 12d362103f9647aca18a71ff40543991 P 4e249f6ef94d4ccc95425ce495376756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3816f29f4663442b96e575d0c48d9dd3" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 35871 } } peers { permanent_uuid: "d9a180d89c524c99919ea0e829d846e4" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 35261 } } peers { permanent_uuid: "4e249f6ef94d4ccc95425ce495376756" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 35563 } }
I20260806 02:49:35.354020 28621 ts_tablet_manager.cc:1428] T 12d362103f9647aca18a71ff40543991 P 3816f29f4663442b96e575d0c48d9dd3: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20260806 02:49:35.354677 28622 ts_tablet_manager.cc:1428] T 12d362103f9647aca18a71ff40543991 P 4e249f6ef94d4ccc95425ce495376756: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260806 02:49:35.354713 28197 heartbeater.cc:503] Master 127.1.183.190:38571 was elected leader, sending a full tablet report...
W20260806 02:49:35.390549 28198 tablet.cc:2369] T 12d362103f9647aca18a71ff40543991 P 4e249f6ef94d4ccc95425ce495376756: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:49:35.392756 28628 raft_consensus.cc:482] T 12d362103f9647aca18a71ff40543991 P 3816f29f4663442b96e575d0c48d9dd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:49:35.392949 28628 raft_consensus.cc:504] T 12d362103f9647aca18a71ff40543991 P 3816f29f4663442b96e575d0c48d9dd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3816f29f4663442b96e575d0c48d9dd3" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 35871 } } peers { permanent_uuid: "d9a180d89c524c99919ea0e829d846e4" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 35261 } } peers { permanent_uuid: "4e249f6ef94d4ccc95425ce495376756" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 35563 } }
I20260806 02:49:35.394099 28628 leader_election.cc:288] T 12d362103f9647aca18a71ff40543991 P 3816f29f4663442b96e575d0c48d9dd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9a180d89c524c99919ea0e829d846e4 (127.1.183.131:35261), 4e249f6ef94d4ccc95425ce495376756 (127.1.183.129:35563)
I20260806 02:49:35.397898 28394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:35.394384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:35261 (local address 127.1.183.130:40405)
0806 02:49:35.394529 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:35.394548 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:35.394651 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:35.394998 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:35.395001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:35.395019 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:35.395165 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:35.395172 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:35.395929 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:35.395934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:35.396785 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:35.396795 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:35.397449 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:35.397453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:35.397559 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:35.397588 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 02:49:35.397674 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 02:49:35.397759 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":1}
I20260806 02:49:35.397898 28616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:35.394604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:40405 (local address 127.1.183.131:35261)
0806 02:49:35.394745 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:35.394749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:35.394766 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:49:35.394833 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:35.394836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:35.394875 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:35.394969 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:35.395301 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:35.395814 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:35.396920 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:35.397324 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:35.397460 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:35.397513 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:35.397704 (+ 191us) server_negotiation.cc:299] Negotiation successful
0806 02:49:35.397773 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:49:35.398386 28618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:35.394918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:60277 (local address 127.1.183.129:35563)
0806 02:49:35.395082 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:35.395087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:35.395252 (+ 165us) server_negotiation.cc:407] Connection header received
0806 02:49:35.395405 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:35.395410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:35.395457 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:35.395542 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:35.395818 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:35.396383 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:35.397416 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:35.397796 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:35.397949 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:35.397996 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:35.398204 (+ 208us) server_negotiation.cc:299] Negotiation successful
0806 02:49:35.398277 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 02:49:35.398413 28631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:35.394695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:35563 (local address 127.1.183.130:60277)
0806 02:49:35.395158 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:35.395169 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:35.395286 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:35.395555 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:35.395558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:35.395575 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:35.395703 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:35.395709 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:35.396499 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:35.396503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:35.397273 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:35.397281 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:35.397960 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:35.397964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:35.398075 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:35.398111 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 02:49:35.398190 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 02:49:35.398254 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":3,"thread_start_us":167,"threads_started":1}
I20260806 02:49:35.398660 28522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12d362103f9647aca18a71ff40543991" candidate_uuid: "3816f29f4663442b96e575d0c48d9dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9a180d89c524c99919ea0e829d846e4" is_pre_election: true
I20260806 02:49:35.399024 28522 raft_consensus.cc:2454] T 12d362103f9647aca18a71ff40543991 P d9a180d89c524c99919ea0e829d846e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3816f29f4663442b96e575d0c48d9dd3 in term 0.
I20260806 02:49:35.399156 28130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12d362103f9647aca18a71ff40543991" candidate_uuid: "3816f29f4663442b96e575d0c48d9dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e249f6ef94d4ccc95425ce495376756" is_pre_election: true
I20260806 02:49:35.399475 28130 raft_consensus.cc:2454] T 12d362103f9647aca18a71ff40543991 P 4e249f6ef94d4ccc95425ce495376756 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3816f29f4663442b96e575d0c48d9dd3 in term 0.
I20260806 02:49:35.399610 28224 leader_election.cc:302] T 12d362103f9647aca18a71ff40543991 P 3816f29f4663442b96e575d0c48d9dd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3816f29f4663442b96e575d0c48d9dd3, d9a180d89c524c99919ea0e829d846e4; no voters:
I20260806 02:49:35.399940 28628 raft_consensus.cc:2781] T 12d362103f9647aca18a71ff40543991 P 3816f29f4663442b96e575d0c48d9dd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:49:35.400038 28628 raft_consensus.cc:482] T 12d362103f9647aca18a71ff40543991 P 3816f29f4663442b96e575d0c48d9dd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:49:35.400193 28628 raft_consensus.cc:3037] T 12d362103f9647aca18a71ff40543991 P 3816f29f4663442b96e575d0c48d9dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:35.401587 28628 raft_consensus.cc:504] T 12d362103f9647aca18a71ff40543991 P 3816f29f4663442b96e575d0c48d9dd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3816f29f4663442b96e575d0c48d9dd3" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 35871 } } peers { permanent_uuid: "d9a180d89c524c99919ea0e829d846e4" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 35261 } } peers { permanent_uuid: "4e249f6ef94d4ccc95425ce495376756" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 35563 } }
I20260806 02:49:35.401985 28628 leader_election.cc:288] T 12d362103f9647aca18a71ff40543991 P 3816f29f4663442b96e575d0c48d9dd3 [CANDIDATE]: Term 1 election: Requested vote from peers d9a180d89c524c99919ea0e829d846e4 (127.1.183.131:35261), 4e249f6ef94d4ccc95425ce495376756 (127.1.183.129:35563)
I20260806 02:49:35.402356 28522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12d362103f9647aca18a71ff40543991" candidate_uuid: "3816f29f4663442b96e575d0c48d9dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9a180d89c524c99919ea0e829d846e4"
I20260806 02:49:35.402432 28130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12d362103f9647aca18a71ff40543991" candidate_uuid: "3816f29f4663442b96e575d0c48d9dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e249f6ef94d4ccc95425ce495376756"
I20260806 02:49:35.402508 28522 raft_consensus.cc:3037] T 12d362103f9647aca18a71ff40543991 P d9a180d89c524c99919ea0e829d846e4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:35.402571 28130 raft_consensus.cc:3037] T 12d362103f9647aca18a71ff40543991 P 4e249f6ef94d4ccc95425ce495376756 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:35.403836 28522 raft_consensus.cc:2454] T 12d362103f9647aca18a71ff40543991 P d9a180d89c524c99919ea0e829d846e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3816f29f4663442b96e575d0c48d9dd3 in term 1.
I20260806 02:49:35.403884 28130 raft_consensus.cc:2454] T 12d362103f9647aca18a71ff40543991 P 4e249f6ef94d4ccc95425ce495376756 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3816f29f4663442b96e575d0c48d9dd3 in term 1.
I20260806 02:49:35.404255 28224 leader_election.cc:302] T 12d362103f9647aca18a71ff40543991 P 3816f29f4663442b96e575d0c48d9dd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3816f29f4663442b96e575d0c48d9dd3, 4e249f6ef94d4ccc95425ce495376756; no voters:
I20260806 02:49:35.404438 28628 raft_consensus.cc:2781] T 12d362103f9647aca18a71ff40543991 P 3816f29f4663442b96e575d0c48d9dd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:49:35.404698 28628 raft_consensus.cc:686] T 12d362103f9647aca18a71ff40543991 P 3816f29f4663442b96e575d0c48d9dd3 [term 1 LEADER]: Becoming Leader. State: Replica: 3816f29f4663442b96e575d0c48d9dd3, State: Running, Role: LEADER
I20260806 02:49:35.405063 28628 consensus_queue.cc:237] T 12d362103f9647aca18a71ff40543991 P 3816f29f4663442b96e575d0c48d9dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3816f29f4663442b96e575d0c48d9dd3" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 35871 } } peers { permanent_uuid: "d9a180d89c524c99919ea0e829d846e4" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 35261 } } peers { permanent_uuid: "4e249f6ef94d4ccc95425ce495376756" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 35563 } }
I20260806 02:49:35.408329 27908 catalog_manager.cc:5495] T 12d362103f9647aca18a71ff40543991 P 3816f29f4663442b96e575d0c48d9dd3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3816f29f4663442b96e575d0c48d9dd3 (127.1.183.130). New cstate: current_term: 1 leader_uuid: "3816f29f4663442b96e575d0c48d9dd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3816f29f4663442b96e575d0c48d9dd3" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 35871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9a180d89c524c99919ea0e829d846e4" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e249f6ef94d4ccc95425ce495376756" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 35563 } health_report { overall_health: UNKNOWN } } }
I20260806 02:49:35.431326 28615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:35.428048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60172 (local address 127.1.183.130:35871)
0806 02:49:35.428263 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:35.428267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:35.428281 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:49:35.428358 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:35.428360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:35.428418 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:35.428507 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:49:35.428717 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:35.429214 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:35.430002 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:35.430202 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:35.430283 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:35.430538 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:49:35.430935 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:49:35.431061 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:35.431096 (+ 35us) server_negotiation.cc:299] Negotiation successful
0806 02:49:35.431159 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 02:49:35.437436 28522 raft_consensus.cc:1261] T 12d362103f9647aca18a71ff40543991 P d9a180d89c524c99919ea0e829d846e4 [term 1 FOLLOWER]: Refusing update from remote peer 3816f29f4663442b96e575d0c48d9dd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:49:35.437453 28130 raft_consensus.cc:1261] T 12d362103f9647aca18a71ff40543991 P 4e249f6ef94d4ccc95425ce495376756 [term 1 FOLLOWER]: Refusing update from remote peer 3816f29f4663442b96e575d0c48d9dd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:49:35.438189 28632 consensus_queue.cc:1035] T 12d362103f9647aca18a71ff40543991 P 3816f29f4663442b96e575d0c48d9dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9a180d89c524c99919ea0e829d846e4" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:49:35.438403 28628 consensus_queue.cc:1035] T 12d362103f9647aca18a71ff40543991 P 3816f29f4663442b96e575d0c48d9dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e249f6ef94d4ccc95425ce495376756" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 35563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:49:35.447971 28639 mvcc.cc:205] Tried to move back new op lower bound from 7315392820983603200 to 7315392820861792256. Current Snapshot: MvccSnapshot[applied={T|T < 7315392820983603200}]
I20260806 02:49:35.454525 27908 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:49:35.454720 27908 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:49:35.457170 27908 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 244a89e1e7284b35b2e764b3ee75f18c: Sending DeleteTablet for 3 replicas of tablet 12d362103f9647aca18a71ff40543991
I20260806 02:49:35.458071 28302 tablet_service.cc:1514] Processing DeleteTablet for tablet 12d362103f9647aca18a71ff40543991 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:49:35 UTC) from {username='slave'} at 127.0.0.1:60158
I20260806 02:49:35.458076 28092 tablet_service.cc:1514] Processing DeleteTablet for tablet 12d362103f9647aca18a71ff40543991 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:49:35 UTC) from {username='slave'} at 127.0.0.1:36146
I20260806 02:49:35.458663 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 28007
I20260806 02:49:35.458616 28494 tablet_service.cc:1514] Processing DeleteTablet for tablet 12d362103f9647aca18a71ff40543991 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:49:35 UTC) from {username='slave'} at 127.0.0.1:45052
I20260806 02:49:35.461148 28646 tablet_replica.cc:330] T 12d362103f9647aca18a71ff40543991 P 3816f29f4663442b96e575d0c48d9dd3: stopping tablet replica
I20260806 02:49:35.461519 28646 raft_consensus.cc:2229] T 12d362103f9647aca18a71ff40543991 P 3816f29f4663442b96e575d0c48d9dd3 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:49:35.461879 28646 raft_consensus.cc:2258] T 12d362103f9647aca18a71ff40543991 P 3816f29f4663442b96e575d0c48d9dd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:49:35.463860 28646 ts_tablet_manager.cc:1905] T 12d362103f9647aca18a71ff40543991 P 3816f29f4663442b96e575d0c48d9dd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:49:35.466866 28646 ts_tablet_manager.cc:1918] T 12d362103f9647aca18a71ff40543991 P 3816f29f4663442b96e575d0c48d9dd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:49:35.466977 28646 log.cc:1196] T 12d362103f9647aca18a71ff40543991 P 3816f29f4663442b96e575d0c48d9dd3: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-1/wal/wals/12d362103f9647aca18a71ff40543991
I20260806 02:49:35.467324 28646 ts_tablet_manager.cc:1939] T 12d362103f9647aca18a71ff40543991 P 3816f29f4663442b96e575d0c48d9dd3: Deleting consensus metadata
I20260806 02:49:35.467561 28648 tablet_replica.cc:330] T 12d362103f9647aca18a71ff40543991 P d9a180d89c524c99919ea0e829d846e4: stopping tablet replica
I20260806 02:49:35.467937 28648 raft_consensus.cc:2229] T 12d362103f9647aca18a71ff40543991 P d9a180d89c524c99919ea0e829d846e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:49:35.468314 28648 raft_consensus.cc:2258] T 12d362103f9647aca18a71ff40543991 P d9a180d89c524c99919ea0e829d846e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:49:35.469931 27889 catalog_manager.cc:4841] TS 3816f29f4663442b96e575d0c48d9dd3 (127.1.183.130:35871): tablet 12d362103f9647aca18a71ff40543991 (table test-table [id=79fe25b940e44ca89c767e739a39ab60]) successfully deleted
I20260806 02:49:35.472290 28648 ts_tablet_manager.cc:1905] T 12d362103f9647aca18a71ff40543991 P d9a180d89c524c99919ea0e829d846e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:49:35.475189 28648 ts_tablet_manager.cc:1918] T 12d362103f9647aca18a71ff40543991 P d9a180d89c524c99919ea0e829d846e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:49:35.475302 28648 log.cc:1196] T 12d362103f9647aca18a71ff40543991 P d9a180d89c524c99919ea0e829d846e4: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984313887056-1758-0/minicluster-data/ts-2/wal/wals/12d362103f9647aca18a71ff40543991
I20260806 02:49:35.475646 28648 ts_tablet_manager.cc:1939] T 12d362103f9647aca18a71ff40543991 P d9a180d89c524c99919ea0e829d846e4: Deleting consensus metadata
W20260806 02:49:35.476724 27890 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.183.129:35563 (error 108)
I20260806 02:49:35.477296 27890 catalog_manager.cc:4841] TS d9a180d89c524c99919ea0e829d846e4 (127.1.183.131:35261): tablet 12d362103f9647aca18a71ff40543991 (table test-table [id=79fe25b940e44ca89c767e739a39ab60]) successfully deleted
I20260806 02:49:35.477830 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 28202
I20260806 02:49:35.477952 28617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:35.477621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:35563 (local address 127.0.0.1:36154)
0806 02:49:35.477777 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:35.477860 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.129:35563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 02:49:35.485350 27890 catalog_manager.cc:4584] TS 4e249f6ef94d4ccc95425ce495376756 (127.1.183.129:35563): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 12d362103f9647aca18a71ff40543991: Network error: Client connection negotiation failed: client connection to 127.1.183.129:35563: connect: Connection refused (error 111)
I20260806 02:49:35.494140 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 28396
I20260806 02:49:35.504380 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 27867
2026-08-06T02:49:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1079 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:49:35 dist-test-slave-gmjp krb5kdc[28654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:49:35 dist-test-slave-gmjp krb5kdc[28654](info): set up 2 sockets
Aug 06 02:49:35 dist-test-slave-gmjp krb5kdc[28654](info): commencing operation
krb5kdc: starting...
W20260806 02:49:36.734347 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.203s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:49:36 dist-test-slave-gmjp krb5kdc[28654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:49:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:49:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
I20260806 02:49:36.865307 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:40269
--webserver_interface=127.1.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:42363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:40269
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:49:36.983577 28671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:36.983896 28671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:36.983971 28671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:36.988070 28671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:49:36.988185 28671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:36.988237 28671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:36.988273 28671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:49:36.988308 28671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:49:36.993652 28671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:42363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:40269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:40269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:36.994776 28671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:36.995832 28671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:37.004964 28678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:37.004964 28680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:37.005144 28671 server_base.cc:1034] running on GCE node
W20260806 02:49:37.005192 28682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:37.005806 28671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:37.006986 28671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:37.008206 28671 hybrid_clock.cc:648] HybridClock initialized: now 1785984577008131 us; error 34 us; skew 500 ppm
Aug 06 02:49:37 dist-test-slave-gmjp krb5kdc[28654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984577, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:37.015548 28671 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:49:37.016793 28671 webserver.cc:460] Webserver started at http://127.1.183.190:45235/ using document root <none> and password file <none>
I20260806 02:49:37.017445 28671 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:37.017513 28671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:37.017761 28671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:37.019956 28671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "7e66769bf6984ec2a382f284d8552228"
format_stamp: "Formatted at 2026-08-06 02:49:37 on dist-test-slave-gmjp"
server_key: "e42febfe04685835a4b86cbb8e9b45ed"
server_key_iv: "bda774fddf6d8341f7e148fbf6c4b961"
server_key_version: "encryptionkey@0"
I20260806 02:49:37.020620 28671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "7e66769bf6984ec2a382f284d8552228"
format_stamp: "Formatted at 2026-08-06 02:49:37 on dist-test-slave-gmjp"
server_key: "e42febfe04685835a4b86cbb8e9b45ed"
server_key_iv: "bda774fddf6d8341f7e148fbf6c4b961"
server_key_version: "encryptionkey@0"
I20260806 02:49:37.024526 28671 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:49:37.026993 28697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:37.028308 28671 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:49:37.028448 28671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "7e66769bf6984ec2a382f284d8552228"
format_stamp: "Formatted at 2026-08-06 02:49:37 on dist-test-slave-gmjp"
server_key: "e42febfe04685835a4b86cbb8e9b45ed"
server_key_iv: "bda774fddf6d8341f7e148fbf6c4b961"
server_key_version: "encryptionkey@0"
I20260806 02:49:37.028564 28671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:37.037563 28671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:37.040450 28671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:37.040653 28671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:37.048094 28671 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:40269
I20260806 02:49:37.048120 28775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:40269 every 8 connection(s)
I20260806 02:49:37.049237 28671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:49:37.052040 28777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:37.053025 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 28671
I20260806 02:49:37.053179 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:49:37.053512 1758 external_mini_cluster.cc:1428] Setting key ce05c1d42e42721f8e924691a4b16fc7
I20260806 02:49:37.059175 28777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e66769bf6984ec2a382f284d8552228: Bootstrap starting.
I20260806 02:49:37.062083 28777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e66769bf6984ec2a382f284d8552228: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:37.062913 28777 log.cc:824] T 00000000000000000000000000000000 P 7e66769bf6984ec2a382f284d8552228: Log is configured to *not* fsync() on all Append() calls
Aug 06 02:49:37 dist-test-slave-gmjp krb5kdc[28654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:49:37.065310 28777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e66769bf6984ec2a382f284d8552228: No bootstrap required, opened a new log
I20260806 02:49:37.068974 28781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:37.055006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34930 (local address 127.1.183.190:40269)
0806 02:49:37.055488 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:37.055508 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:37.055556 (+ 48us) server_negotiation.cc:407] Connection header received
0806 02:49:37.056444 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:37.056468 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:37.056897 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:37.057279 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:37.058023 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.059513 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.060462 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.061188 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.061350 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:37.064114 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:37.064119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:37.064135 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:37.064214 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:37.065982 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:37.066477 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:37.066483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:37.066488 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:37.066598 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:37.066893 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:37.066900 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:37.066903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:37.067303 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:37.067522 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:37.067826 (+ 304us) server_negotiation.cc:299] Negotiation successful
0806 02:49:37.068140 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":114,"threads_started":1}
I20260806 02:49:37.068964 28777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e66769bf6984ec2a382f284d8552228 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e66769bf6984ec2a382f284d8552228" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 40269 } }
I20260806 02:49:37.069239 28777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e66769bf6984ec2a382f284d8552228 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:37.069285 28777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e66769bf6984ec2a382f284d8552228 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e66769bf6984ec2a382f284d8552228, State: Initialized, Role: FOLLOWER
I20260806 02:49:37.069695 28777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e66769bf6984ec2a382f284d8552228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e66769bf6984ec2a382f284d8552228" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 40269 } }
I20260806 02:49:37.069802 28777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e66769bf6984ec2a382f284d8552228 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:49:37.069841 28777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e66769bf6984ec2a382f284d8552228 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:49:37.069903 28777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e66769bf6984ec2a382f284d8552228 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:37.071033 28777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e66769bf6984ec2a382f284d8552228 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e66769bf6984ec2a382f284d8552228" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 40269 } }
I20260806 02:49:37.071496 28777 leader_election.cc:302] T 00000000000000000000000000000000 P 7e66769bf6984ec2a382f284d8552228 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e66769bf6984ec2a382f284d8552228; no voters:
I20260806 02:49:37.071836 28777 leader_election.cc:288] T 00000000000000000000000000000000 P 7e66769bf6984ec2a382f284d8552228 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:49:37.072005 28784 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e66769bf6984ec2a382f284d8552228 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:49:37.072319 28784 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e66769bf6984ec2a382f284d8552228 [term 1 LEADER]: Becoming Leader. State: Replica: 7e66769bf6984ec2a382f284d8552228, State: Running, Role: LEADER
I20260806 02:49:37.072642 28784 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e66769bf6984ec2a382f284d8552228 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e66769bf6984ec2a382f284d8552228" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 40269 } }
I20260806 02:49:37.073180 28777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e66769bf6984ec2a382f284d8552228 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:49:37.074270 28786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e66769bf6984ec2a382f284d8552228 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e66769bf6984ec2a382f284d8552228. Latest consensus state: current_term: 1 leader_uuid: "7e66769bf6984ec2a382f284d8552228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e66769bf6984ec2a382f284d8552228" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 40269 } } }
I20260806 02:49:37.074404 28786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e66769bf6984ec2a382f284d8552228 [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:37.074761 28785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e66769bf6984ec2a382f284d8552228 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e66769bf6984ec2a382f284d8552228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e66769bf6984ec2a382f284d8552228" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 40269 } } }
I20260806 02:49:37.074862 28785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e66769bf6984ec2a382f284d8552228 [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:37.075150 28797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:49:37.078470 28797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:49:37.084311 28797 catalog_manager.cc:1347] Generated new cluster ID: 2968210461de4c49900ecd9d0ef23e6f
I20260806 02:49:37.084429 28797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:49:37.095427 28797 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:49:37.096685 28797 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:49:37.112541 28797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e66769bf6984ec2a382f284d8552228: Generated new TSK 0
I20260806 02:49:37.113442 28797 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
I20260806 02:49:37.147204 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.129:0
--local_ip_for_outbound_sockets=127.1.183.129
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:40269
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:42363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:49:37.268206 28810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:37.268529 28810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:37.268606 28810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:37.272951 28810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:37.273044 28810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:37.273132 28810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.129
I20260806 02:49:37.278962 28810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:42363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:40269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.129
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:37.280140 28810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:37.281328 28810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:37.291487 28817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:37.291492 28821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:37.291520 28816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:37.291817 28810 server_base.cc:1034] running on GCE node
I20260806 02:49:37.292448 28810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:37.293131 28810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:37.294355 28810 hybrid_clock.cc:648] HybridClock initialized: now 1785984577294333 us; error 35 us; skew 500 ppm
Aug 06 02:49:37 dist-test-slave-gmjp krb5kdc[28654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984577, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:37.297627 28810 init.cc:377] Logged in from keytab as kudu/127.1.183.129@KRBTEST.COM (short username kudu)
I20260806 02:49:37.298666 28810 webserver.cc:460] Webserver started at http://127.1.183.129:33015/ using document root <none> and password file <none>
I20260806 02:49:37.299286 28810 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:37.299391 28810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:37.299643 28810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:37.301895 28810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-0/data/instance:
uuid: "91889189682a4986a022a0a72d373c1b"
format_stamp: "Formatted at 2026-08-06 02:49:37 on dist-test-slave-gmjp"
server_key: "122d9e0c56e957684fca8418c94f5d96"
server_key_iv: "d4aaba73bbd48cc17a433351c641582f"
server_key_version: "encryptionkey@0"
I20260806 02:49:37.302510 28810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance:
uuid: "91889189682a4986a022a0a72d373c1b"
format_stamp: "Formatted at 2026-08-06 02:49:37 on dist-test-slave-gmjp"
server_key: "122d9e0c56e957684fca8418c94f5d96"
server_key_iv: "d4aaba73bbd48cc17a433351c641582f"
server_key_version: "encryptionkey@0"
I20260806 02:49:37.306545 28810 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:49:37.309288 28828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:37.310626 28810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:49:37.310842 28810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-0/wal
uuid: "91889189682a4986a022a0a72d373c1b"
format_stamp: "Formatted at 2026-08-06 02:49:37 on dist-test-slave-gmjp"
server_key: "122d9e0c56e957684fca8418c94f5d96"
server_key_iv: "d4aaba73bbd48cc17a433351c641582f"
server_key_version: "encryptionkey@0"
I20260806 02:49:37.310974 28810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:37.332692 28810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:37.335986 28810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:37.336239 28810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:37.336980 28810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:49:37.337908 28810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:49:37.337961 28810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:37.338037 28810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:49:37.338074 28810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:37.347492 28810 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.129:35933
I20260806 02:49:37.347523 28991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.129:35933 every 8 connection(s)
I20260806 02:49:37.348450 28810 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
I20260806 02:49:37.355252 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 28810
I20260806 02:49:37.355384 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance
I20260806 02:49:37.355695 1758 external_mini_cluster.cc:1428] Setting key 3807b4267cc37d4265e0ae32e36577bc
Aug 06 02:49:37 dist-test-slave-gmjp krb5kdc[28654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984577, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:49:37.362978 28781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:37.350355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:38085 (local address 127.1.183.190:40269)
0806 02:49:37.350545 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:37.350550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:37.351640 (+ 1090us) server_negotiation.cc:407] Connection header received
0806 02:49:37.352912 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:37.352916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:37.352997 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:37.353084 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:37.354314 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.354869 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.356263 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.356606 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.356736 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:37.359259 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:37.359263 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:37.359266 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:37.359292 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:37.360834 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:37.361369 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:37.361373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:37.361375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:37.361460 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:37.361818 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:37.361822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:37.361824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:37.362021 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:37.362135 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:37.362651 (+ 516us) server_negotiation.cc:299] Negotiation successful
0806 02:49:37.362791 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 02:49:37.363791 28994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:37.350795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:40269 (local address 127.1.183.129:38085)
0806 02:49:37.351445 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:37.351505 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:37.352640 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:37.353302 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:37.353309 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:37.353675 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:49:37.354157 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:37.354171 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.355013 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.355020 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:37.356110 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:37.356120 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.356771 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.356776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:37.357012 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:37.357622 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:49:37.357638 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:49:37.359091 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:49:37.360971 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:37.360977 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:37.360992 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:37.361259 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:37.361567 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:37.361570 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:37.361572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:37.361716 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:37.362135 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:49:37.362143 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:49:37.362403 (+ 260us) client_negotiation.cc:769] Sending connection context
0806 02:49:37.362606 (+ 203us) client_negotiation.cc:240] Negotiation successful
0806 02:49:37.362942 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":160,"threads_started":1}
I20260806 02:49:37.365132 28992 heartbeater.cc:347] Connected to a master server at 127.1.183.190:40269
I20260806 02:49:37.365419 28992 heartbeater.cc:464] Registering TS with master...
I20260806 02:49:37.366062 28992 heartbeater.cc:511] Master 127.1.183.190:40269 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.183.130@KRBTEST.COM; defaulting to no policy
I20260806 02:49:37.367718 28720 ts_manager.cc:194] Registered new tserver with Master: 91889189682a4986a022a0a72d373c1b (127.1.183.129:35933)
I20260806 02:49:37.369107 28720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.129@KRBTEST.COM'} at 127.1.183.129:38085
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
I20260806 02:49:37.387614 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.130:0
--local_ip_for_outbound_sockets=127.1.183.130
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:40269
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:42363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:49:37.503813 28998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:37.504143 28998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:37.504233 28998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:37.508306 28998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:37.508414 28998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:37.508512 28998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.130
I20260806 02:49:37.513815 28998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:42363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:40269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.130
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:37.514973 28998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:37.516189 28998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:37.526261 29008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:37.526261 29011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:37.526630 28998 server_base.cc:1034] running on GCE node
W20260806 02:49:37.526280 29009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:37.527104 28998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:37.527751 28998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:37.528926 28998 hybrid_clock.cc:648] HybridClock initialized: now 1785984577528892 us; error 44 us; skew 500 ppm
Aug 06 02:49:37 dist-test-slave-gmjp krb5kdc[28654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984577, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:37.531663 28998 init.cc:377] Logged in from keytab as kudu/127.1.183.130@KRBTEST.COM (short username kudu)
I20260806 02:49:37.532687 28998 webserver.cc:460] Webserver started at http://127.1.183.130:35235/ using document root <none> and password file <none>
I20260806 02:49:37.533224 28998 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:37.533306 28998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:37.533551 28998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:37.535624 28998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-1/data/instance:
uuid: "f43544884dd7407fb8265a8dd63fb9f3"
format_stamp: "Formatted at 2026-08-06 02:49:37 on dist-test-slave-gmjp"
server_key: "b1d2a58434dfc37cdb3a6f3d1b76af39"
server_key_iv: "2903c65ac22f49ff1bd443c9b43b1f49"
server_key_version: "encryptionkey@0"
I20260806 02:49:37.536231 28998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance:
uuid: "f43544884dd7407fb8265a8dd63fb9f3"
format_stamp: "Formatted at 2026-08-06 02:49:37 on dist-test-slave-gmjp"
server_key: "b1d2a58434dfc37cdb3a6f3d1b76af39"
server_key_iv: "2903c65ac22f49ff1bd443c9b43b1f49"
server_key_version: "encryptionkey@0"
I20260806 02:49:37.540028 28998 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260806 02:49:37.542676 29020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:37.544045 28998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:49:37.544194 28998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-1/wal
uuid: "f43544884dd7407fb8265a8dd63fb9f3"
format_stamp: "Formatted at 2026-08-06 02:49:37 on dist-test-slave-gmjp"
server_key: "b1d2a58434dfc37cdb3a6f3d1b76af39"
server_key_iv: "2903c65ac22f49ff1bd443c9b43b1f49"
server_key_version: "encryptionkey@0"
I20260806 02:49:37.544312 28998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:37.558101 28998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:37.561240 28998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:37.561477 28998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:37.562137 28998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:49:37.563007 28998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:49:37.563055 28998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:37.563124 28998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:49:37.563160 28998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:37.572371 28998 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.130:33465
I20260806 02:49:37.572441 29177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.130:33465 every 8 connection(s)
I20260806 02:49:37.573524 28998 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
I20260806 02:49:37.575610 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 28998
I20260806 02:49:37.575748 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance
I20260806 02:49:37.576063 1758 external_mini_cluster.cc:1428] Setting key 9bf88fae1ef5e956f1104517315c8513
Aug 06 02:49:37 dist-test-slave-gmjp krb5kdc[28654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984577, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.183.131@KRBTEST.COM; defaulting to no policy
I20260806 02:49:37.588755 28781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:37.575869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:57469 (local address 127.1.183.190:40269)
0806 02:49:37.576002 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:37.576006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:37.577108 (+ 1102us) server_negotiation.cc:407] Connection header received
0806 02:49:37.578235 (+ 1127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:37.578238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:37.578296 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:37.578376 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:37.579638 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.580238 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.581413 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.581656 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.581773 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:37.584373 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:37.584377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:37.584380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:37.584416 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:37.586393 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:37.586990 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:37.586993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:37.586996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:37.587086 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:37.587510 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:37.587514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:37.587516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:37.587718 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:37.587835 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:37.588440 (+ 605us) server_negotiation.cc:299] Negotiation successful
0806 02:49:37.588579 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.131@KRBTEST.COM" created.
I20260806 02:49:37.589988 29182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:37.576331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:40269 (local address 127.1.183.130:57469)
0806 02:49:37.576911 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:37.576983 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:37.578019 (+ 1036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:37.578581 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:37.578588 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:37.578966 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:49:37.579484 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:37.579499 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.580375 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.580379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:37.581278 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:37.581288 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.581820 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.581824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:37.582003 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:37.582647 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:49:37.582664 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:49:37.584143 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:49:37.586528 (+ 2385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:37.586534 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:37.586551 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:37.586864 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:37.587186 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:37.587189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:37.587191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:37.587346 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:37.587836 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:49:37.587846 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:49:37.588123 (+ 277us) client_negotiation.cc:769] Sending connection context
0806 02:49:37.588387 (+ 264us) client_negotiation.cc:240] Negotiation successful
0806 02:49:37.588835 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":154,"threads_started":1}
I20260806 02:49:37.591264 29178 heartbeater.cc:347] Connected to a master server at 127.1.183.190:40269
I20260806 02:49:37.591555 29178 heartbeater.cc:464] Registering TS with master...
I20260806 02:49:37.592092 29178 heartbeater.cc:511] Master 127.1.183.190:40269 requested a full tablet report, sending...
I20260806 02:49:37.593263 28720 ts_manager.cc:194] Registered new tserver with Master: f43544884dd7407fb8265a8dd63fb9f3 (127.1.183.130:33465)
I20260806 02:49:37.593864 28720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.130@KRBTEST.COM'} at 127.1.183.130:57469
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
I20260806 02:49:37.607750 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.131:0
--local_ip_for_outbound_sockets=127.1.183.131
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:40269
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:42363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:49:37.724239 29185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:37.724566 29185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:37.724655 29185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:37.728782 29185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:37.728885 29185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:37.728983 29185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.131
I20260806 02:49:37.734445 29185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:42363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:40269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.131
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:37.735597 29185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:37.736745 29185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:37.746697 29197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:37.746691 29195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:37.746691 29194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:37.746758 29185 server_base.cc:1034] running on GCE node
I20260806 02:49:37.747283 29185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:37.747979 29185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:37.749171 29185 hybrid_clock.cc:648] HybridClock initialized: now 1785984577749151 us; error 39 us; skew 500 ppm
Aug 06 02:49:37 dist-test-slave-gmjp krb5kdc[28654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984577, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:37.751924 29185 init.cc:377] Logged in from keytab as kudu/127.1.183.131@KRBTEST.COM (short username kudu)
I20260806 02:49:37.752861 29185 webserver.cc:460] Webserver started at http://127.1.183.131:40251/ using document root <none> and password file <none>
I20260806 02:49:37.753446 29185 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:37.753540 29185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:37.753774 29185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:37.755887 29185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-2/data/instance:
uuid: "9e19305103bc47a787612cf54d696491"
format_stamp: "Formatted at 2026-08-06 02:49:37 on dist-test-slave-gmjp"
server_key: "0291307677b3bffb12247c46b3868033"
server_key_iv: "395e6eda9129009112f0aa973648ae77"
server_key_version: "encryptionkey@0"
I20260806 02:49:37.756479 29185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance:
uuid: "9e19305103bc47a787612cf54d696491"
format_stamp: "Formatted at 2026-08-06 02:49:37 on dist-test-slave-gmjp"
server_key: "0291307677b3bffb12247c46b3868033"
server_key_iv: "395e6eda9129009112f0aa973648ae77"
server_key_version: "encryptionkey@0"
I20260806 02:49:37.760147 29185 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:49:37.762540 29204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:37.763691 29185 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:49:37.763835 29185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-2/wal
uuid: "9e19305103bc47a787612cf54d696491"
format_stamp: "Formatted at 2026-08-06 02:49:37 on dist-test-slave-gmjp"
server_key: "0291307677b3bffb12247c46b3868033"
server_key_iv: "395e6eda9129009112f0aa973648ae77"
server_key_version: "encryptionkey@0"
I20260806 02:49:37.763947 29185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:37.775583 29185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:37.778926 29185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:37.779139 29185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:37.779750 29185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:49:37.780664 29185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:49:37.780735 29185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:37.780804 29185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:49:37.780853 29185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:37.789955 29185 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.131:41723
I20260806 02:49:37.789986 29362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.131:41723 every 8 connection(s)
I20260806 02:49:37.790903 29185 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
I20260806 02:49:37.795717 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 29185
I20260806 02:49:37.795873 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance
I20260806 02:49:37.796218 1758 external_mini_cluster.cc:1428] Setting key 28bb1a5c5d9995d1380e566c99acaa19
Aug 06 02:49:37 dist-test-slave-gmjp krb5kdc[28654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984577, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:49:37.805753 28781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:37.792917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:52097 (local address 127.1.183.190:40269)
0806 02:49:37.793096 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:37.793102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:37.794282 (+ 1180us) server_negotiation.cc:407] Connection header received
0806 02:49:37.795195 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:37.795198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:37.795246 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:37.795321 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:37.796963 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.797710 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.799137 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.799348 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.799439 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:37.802243 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:37.802246 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:37.802248 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:37.802273 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:37.803751 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:37.804426 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:37.804428 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:37.804430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:37.804507 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:37.804812 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:37.804815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:37.804816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:37.804995 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:37.805071 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:37.805531 (+ 460us) server_negotiation.cc:299] Negotiation successful
0806 02:49:37.805631 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:49:37.806540 29369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:37.793311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:40269 (local address 127.1.183.131:52097)
0806 02:49:37.794116 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:37.794163 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:37.795008 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:37.795515 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:37.795522 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:37.796085 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:49:37.796776 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:37.796796 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.797865 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.797868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:37.799020 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:37.799030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.799494 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.799497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:37.799645 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:37.800320 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:49:37.800338 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:49:37.802026 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:49:37.803955 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:37.803961 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:37.803973 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:37.804323 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:37.804593 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:37.804595 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:37.804596 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:37.804726 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:37.805085 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:49:37.805090 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:49:37.805321 (+ 231us) client_negotiation.cc:769] Sending connection context
0806 02:49:37.805469 (+ 148us) client_negotiation.cc:240] Negotiation successful
0806 02:49:37.805713 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":121,"threads_started":1}
I20260806 02:49:37.807646 29363 heartbeater.cc:347] Connected to a master server at 127.1.183.190:40269
I20260806 02:49:37.807904 29363 heartbeater.cc:464] Registering TS with master...
I20260806 02:49:37.808481 29363 heartbeater.cc:511] Master 127.1.183.190:40269 requested a full tablet report, sending...
I20260806 02:49:37.809677 28720 ts_manager.cc:194] Registered new tserver with Master: 9e19305103bc47a787612cf54d696491 (127.1.183.131:41723)
I20260806 02:49:37.810290 1758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:49:37.810345 28720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.131@KRBTEST.COM'} at 127.1.183.131:52097
I20260806 02:49:37.820533 28781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:37.812357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34940 (local address 127.1.183.190:40269)
0806 02:49:37.812487 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:37.812493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:37.812742 (+ 249us) server_negotiation.cc:407] Connection header received
0806 02:49:37.812817 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:37.812819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:37.812858 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:37.812928 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:37.813697 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.814269 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.815259 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.815469 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.815551 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:37.817251 (+ 1700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:37.817255 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:37.817257 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:37.817280 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:37.818596 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:37.819084 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:37.819087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:37.819088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:37.819159 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:37.819429 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:37.819431 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:37.819432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:37.819642 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:37.819757 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:37.820144 (+ 387us) server_negotiation.cc:299] Negotiation successful
0806 02:49:37.820346 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 02:49:37.823710 28720 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:49:37.826951 28720 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:49:37.843426 29387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:37.837502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:35933 (local address 127.0.0.1:56380)
0806 02:49:37.838318 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:37.838333 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:37.838461 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:37.839010 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:37.839016 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:37.839059 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:37.839297 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:37.839305 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.840818 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.840821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:37.841816 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:37.841827 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.842797 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.842801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:37.842937 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:37.842986 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 02:49:37.843108 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 02:49:37.843239 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":38,"spinlock_wait_cycles":13824,"thread_start_us":92,"threads_started":1}
I20260806 02:49:37.843461 29389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:37.837904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56380 (local address 127.1.183.129:35933)
0806 02:49:37.838236 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:37.838253 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:37.838437 (+ 184us) server_negotiation.cc:407] Connection header received
0806 02:49:37.838588 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:37.838595 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:37.838807 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:37.839183 (+ 376us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:37.839413 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.840662 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.841964 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.842669 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.842784 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:37.842946 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:37.843170 (+ 224us) server_negotiation.cc:299] Negotiation successful
0806 02:49:37.843302 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":113,"threads_started":1}
I20260806 02:49:37.844280 29379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:37.836301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:33465 (local address 127.0.0.1:48770)
0806 02:49:37.836678 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:37.836728 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:37.836868 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:37.839012 (+ 2144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:37.839019 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:37.839068 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:37.839347 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:37.839357 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.840860 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.840866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:37.842434 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:37.842446 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.843787 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.843791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:37.843900 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:37.843929 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 02:49:37.844012 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 02:49:37.844099 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":205,"spinlock_wait_cycles":10496,"thread_start_us":104,"threads_started":1}
I20260806 02:49:37.844991 29382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:37.836426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48770 (local address 127.1.183.130:33465)
0806 02:49:37.837016 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:37.837029 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:37.837058 (+ 29us) server_negotiation.cc:407] Connection header received
0806 02:49:37.837140 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:37.837148 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:37.837386 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:37.837571 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:37.839462 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.840664 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.842594 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.843445 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.843563 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:37.843726 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:37.844716 (+ 990us) server_negotiation.cc:299] Negotiation successful
0806 02:49:37.844836 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":176,"threads_started":1}
I20260806 02:49:37.845086 29385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:37.837338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45148 (local address 127.1.183.131:41723)
0806 02:49:37.838732 (+ 1394us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:37.838744 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:37.838766 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:49:37.838839 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:37.838844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:37.839022 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:37.839181 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:37.839757 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.840809 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.842577 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.844420 (+ 1843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.844535 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:37.844658 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:37.844830 (+ 172us) server_negotiation.cc:299] Negotiation successful
0806 02:49:37.844946 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1289,"thread_start_us":143,"threads_started":1}
I20260806 02:49:37.846880 28898 tablet_service.cc:1467] Processing CreateTablet for tablet c8d21aa5e17345d685f6f66e56047785 (DEFAULT_TABLE table=test-table [id=c1413e51fdf242d0b4de876e8bcb737b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:49:37.847391 29384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:37.837156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:41723 (local address 127.0.0.1:45148)
0806 02:49:37.837603 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:37.837618 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:37.837708 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:37.839154 (+ 1446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:37.839157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:37.839171 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:37.839416 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:37.839422 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.840949 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.840952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:37.842434 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:37.842446 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.844556 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.844560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:37.844666 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:37.844688 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:49:37.844786 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 02:49:37.847255 (+ 2469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":143,"thread_start_us":95,"threads_started":1}
I20260806 02:49:37.847487 29089 tablet_service.cc:1467] Processing CreateTablet for tablet c8d21aa5e17345d685f6f66e56047785 (DEFAULT_TABLE table=test-table [id=c1413e51fdf242d0b4de876e8bcb737b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:49:37.848217 28898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8d21aa5e17345d685f6f66e56047785. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:37.848712 29089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8d21aa5e17345d685f6f66e56047785. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:37.850911 29275 tablet_service.cc:1467] Processing CreateTablet for tablet c8d21aa5e17345d685f6f66e56047785 (DEFAULT_TABLE table=test-table [id=c1413e51fdf242d0b4de876e8bcb737b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:49:37.851992 29275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8d21aa5e17345d685f6f66e56047785. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:37.855189 29390 tablet_bootstrap.cc:492] T c8d21aa5e17345d685f6f66e56047785 P 91889189682a4986a022a0a72d373c1b: Bootstrap starting.
I20260806 02:49:37.856794 29391 tablet_bootstrap.cc:492] T c8d21aa5e17345d685f6f66e56047785 P f43544884dd7407fb8265a8dd63fb9f3: Bootstrap starting.
I20260806 02:49:37.857600 29390 tablet_bootstrap.cc:654] T c8d21aa5e17345d685f6f66e56047785 P 91889189682a4986a022a0a72d373c1b: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:37.858995 29390 log.cc:824] T c8d21aa5e17345d685f6f66e56047785 P 91889189682a4986a022a0a72d373c1b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:37.859144 29391 tablet_bootstrap.cc:654] T c8d21aa5e17345d685f6f66e56047785 P f43544884dd7407fb8265a8dd63fb9f3: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:37.860345 29391 log.cc:824] T c8d21aa5e17345d685f6f66e56047785 P f43544884dd7407fb8265a8dd63fb9f3: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:37.861269 29390 tablet_bootstrap.cc:492] T c8d21aa5e17345d685f6f66e56047785 P 91889189682a4986a022a0a72d373c1b: No bootstrap required, opened a new log
I20260806 02:49:37.861497 29390 ts_tablet_manager.cc:1397] T c8d21aa5e17345d685f6f66e56047785 P 91889189682a4986a022a0a72d373c1b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:49:37.862092 29391 tablet_bootstrap.cc:492] T c8d21aa5e17345d685f6f66e56047785 P f43544884dd7407fb8265a8dd63fb9f3: No bootstrap required, opened a new log
I20260806 02:49:37.862298 29391 ts_tablet_manager.cc:1397] T c8d21aa5e17345d685f6f66e56047785 P f43544884dd7407fb8265a8dd63fb9f3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:49:37.864562 29390 raft_consensus.cc:348] T c8d21aa5e17345d685f6f66e56047785 P 91889189682a4986a022a0a72d373c1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f43544884dd7407fb8265a8dd63fb9f3" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33465 } } peers { permanent_uuid: "9e19305103bc47a787612cf54d696491" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 41723 } } peers { permanent_uuid: "91889189682a4986a022a0a72d373c1b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 35933 } }
I20260806 02:49:37.864812 29390 raft_consensus.cc:374] T c8d21aa5e17345d685f6f66e56047785 P 91889189682a4986a022a0a72d373c1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:37.864849 29392 tablet_bootstrap.cc:492] T c8d21aa5e17345d685f6f66e56047785 P 9e19305103bc47a787612cf54d696491: Bootstrap starting.
I20260806 02:49:37.864905 29390 raft_consensus.cc:729] T c8d21aa5e17345d685f6f66e56047785 P 91889189682a4986a022a0a72d373c1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91889189682a4986a022a0a72d373c1b, State: Initialized, Role: FOLLOWER
I20260806 02:49:37.865420 29390 consensus_queue.cc:260] T c8d21aa5e17345d685f6f66e56047785 P 91889189682a4986a022a0a72d373c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f43544884dd7407fb8265a8dd63fb9f3" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33465 } } peers { permanent_uuid: "9e19305103bc47a787612cf54d696491" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 41723 } } peers { permanent_uuid: "91889189682a4986a022a0a72d373c1b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 35933 } }
I20260806 02:49:37.865439 29391 raft_consensus.cc:348] T c8d21aa5e17345d685f6f66e56047785 P f43544884dd7407fb8265a8dd63fb9f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f43544884dd7407fb8265a8dd63fb9f3" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33465 } } peers { permanent_uuid: "9e19305103bc47a787612cf54d696491" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 41723 } } peers { permanent_uuid: "91889189682a4986a022a0a72d373c1b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 35933 } }
I20260806 02:49:37.865805 29391 raft_consensus.cc:374] T c8d21aa5e17345d685f6f66e56047785 P f43544884dd7407fb8265a8dd63fb9f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:37.865909 29391 raft_consensus.cc:729] T c8d21aa5e17345d685f6f66e56047785 P f43544884dd7407fb8265a8dd63fb9f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f43544884dd7407fb8265a8dd63fb9f3, State: Initialized, Role: FOLLOWER
I20260806 02:49:37.866422 29391 consensus_queue.cc:260] T c8d21aa5e17345d685f6f66e56047785 P f43544884dd7407fb8265a8dd63fb9f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f43544884dd7407fb8265a8dd63fb9f3" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33465 } } peers { permanent_uuid: "9e19305103bc47a787612cf54d696491" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 41723 } } peers { permanent_uuid: "91889189682a4986a022a0a72d373c1b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 35933 } }
I20260806 02:49:37.866568 29390 ts_tablet_manager.cc:1428] T c8d21aa5e17345d685f6f66e56047785 P 91889189682a4986a022a0a72d373c1b: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260806 02:49:37.866942 28992 heartbeater.cc:503] Master 127.1.183.190:40269 was elected leader, sending a full tablet report...
I20260806 02:49:37.867095 29178 heartbeater.cc:503] Master 127.1.183.190:40269 was elected leader, sending a full tablet report...
I20260806 02:49:37.867381 29392 tablet_bootstrap.cc:654] T c8d21aa5e17345d685f6f66e56047785 P 9e19305103bc47a787612cf54d696491: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:37.867408 29391 ts_tablet_manager.cc:1428] T c8d21aa5e17345d685f6f66e56047785 P f43544884dd7407fb8265a8dd63fb9f3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 02:49:37.868811 29392 log.cc:824] T c8d21aa5e17345d685f6f66e56047785 P 9e19305103bc47a787612cf54d696491: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:37.871151 29392 tablet_bootstrap.cc:492] T c8d21aa5e17345d685f6f66e56047785 P 9e19305103bc47a787612cf54d696491: No bootstrap required, opened a new log
I20260806 02:49:37.871459 29392 ts_tablet_manager.cc:1397] T c8d21aa5e17345d685f6f66e56047785 P 9e19305103bc47a787612cf54d696491: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.004s
I20260806 02:49:37.874946 29392 raft_consensus.cc:348] T c8d21aa5e17345d685f6f66e56047785 P 9e19305103bc47a787612cf54d696491 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f43544884dd7407fb8265a8dd63fb9f3" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33465 } } peers { permanent_uuid: "9e19305103bc47a787612cf54d696491" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 41723 } } peers { permanent_uuid: "91889189682a4986a022a0a72d373c1b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 35933 } }
I20260806 02:49:37.875201 29392 raft_consensus.cc:374] T c8d21aa5e17345d685f6f66e56047785 P 9e19305103bc47a787612cf54d696491 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:37.875290 29392 raft_consensus.cc:729] T c8d21aa5e17345d685f6f66e56047785 P 9e19305103bc47a787612cf54d696491 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e19305103bc47a787612cf54d696491, State: Initialized, Role: FOLLOWER
I20260806 02:49:37.875762 29392 consensus_queue.cc:260] T c8d21aa5e17345d685f6f66e56047785 P 9e19305103bc47a787612cf54d696491 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f43544884dd7407fb8265a8dd63fb9f3" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33465 } } peers { permanent_uuid: "9e19305103bc47a787612cf54d696491" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 41723 } } peers { permanent_uuid: "91889189682a4986a022a0a72d373c1b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 35933 } }
I20260806 02:49:37.876542 29363 heartbeater.cc:503] Master 127.1.183.190:40269 was elected leader, sending a full tablet report...
I20260806 02:49:37.876816 29392 ts_tablet_manager.cc:1428] T c8d21aa5e17345d685f6f66e56047785 P 9e19305103bc47a787612cf54d696491: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20260806 02:49:37.894097 29399 raft_consensus.cc:482] T c8d21aa5e17345d685f6f66e56047785 P 9e19305103bc47a787612cf54d696491 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:49:37.894335 29399 raft_consensus.cc:504] T c8d21aa5e17345d685f6f66e56047785 P 9e19305103bc47a787612cf54d696491 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f43544884dd7407fb8265a8dd63fb9f3" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33465 } } peers { permanent_uuid: "9e19305103bc47a787612cf54d696491" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 41723 } } peers { permanent_uuid: "91889189682a4986a022a0a72d373c1b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 35933 } }
I20260806 02:49:37.895509 29399 leader_election.cc:288] T c8d21aa5e17345d685f6f66e56047785 P 9e19305103bc47a787612cf54d696491 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f43544884dd7407fb8265a8dd63fb9f3 (127.1.183.130:33465), 91889189682a4986a022a0a72d373c1b (127.1.183.129:35933)
I20260806 02:49:37.899708 29369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:37.895779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:33465 (local address 127.1.183.131:51967)
0806 02:49:37.895914 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:37.895931 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:37.896035 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:37.896387 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:37.896391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:37.896411 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:37.896536 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:37.896543 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.897470 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.897474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:37.898631 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:37.898643 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.899309 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.899311 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:37.899465 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:37.899486 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 02:49:37.899547 (+ 61us) client_negotiation.cc:240] Negotiation successful
0806 02:49:37.899602 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 02:49:37.899784 29382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:37.895941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:51967 (local address 127.1.183.130:33465)
0806 02:49:37.896102 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:37.896108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:37.896124 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:49:37.896206 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:37.896210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:37.896267 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:37.896382 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:37.896675 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.897345 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.898761 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.899198 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.899316 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:37.899380 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:37.899588 (+ 208us) server_negotiation.cc:299] Negotiation successful
0806 02:49:37.899659 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:49:37.900467 29400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:37.896110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:35933 (local address 127.1.183.131:58921)
0806 02:49:37.896593 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:37.896605 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:37.896722 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:37.896972 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:37.896974 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:37.896991 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:37.897119 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:37.897126 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.898047 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.898051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:37.899067 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:37.899075 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.900042 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.900046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:37.900144 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:37.900190 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 02:49:37.900270 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 02:49:37.900337 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":143,"threads_started":1}
I20260806 02:49:37.900483 29389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:37.896290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:58921 (local address 127.1.183.129:35933)
0806 02:49:37.896445 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:37.896453 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:37.896685 (+ 232us) server_negotiation.cc:407] Connection header received
0806 02:49:37.896812 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:37.896815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:37.896870 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:37.896960 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:37.897246 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.897910 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.899187 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.899753 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.899888 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:37.899952 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:37.900299 (+ 347us) server_negotiation.cc:299] Negotiation successful
0806 02:49:37.900379 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:49:37.900458 29115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d21aa5e17345d685f6f66e56047785" candidate_uuid: "9e19305103bc47a787612cf54d696491" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f43544884dd7407fb8265a8dd63fb9f3" is_pre_election: true
I20260806 02:49:37.900781 29115 raft_consensus.cc:2454] T c8d21aa5e17345d685f6f66e56047785 P f43544884dd7407fb8265a8dd63fb9f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e19305103bc47a787612cf54d696491 in term 0.
I20260806 02:49:37.901108 28927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d21aa5e17345d685f6f66e56047785" candidate_uuid: "9e19305103bc47a787612cf54d696491" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91889189682a4986a022a0a72d373c1b" is_pre_election: true
I20260806 02:49:37.901302 29208 leader_election.cc:302] T c8d21aa5e17345d685f6f66e56047785 P 9e19305103bc47a787612cf54d696491 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e19305103bc47a787612cf54d696491, f43544884dd7407fb8265a8dd63fb9f3; no voters:
I20260806 02:49:37.901379 28927 raft_consensus.cc:2454] T c8d21aa5e17345d685f6f66e56047785 P 91889189682a4986a022a0a72d373c1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e19305103bc47a787612cf54d696491 in term 0.
I20260806 02:49:37.901594 29399 raft_consensus.cc:2781] T c8d21aa5e17345d685f6f66e56047785 P 9e19305103bc47a787612cf54d696491 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:49:37.901692 29399 raft_consensus.cc:482] T c8d21aa5e17345d685f6f66e56047785 P 9e19305103bc47a787612cf54d696491 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:49:37.901743 29399 raft_consensus.cc:3037] T c8d21aa5e17345d685f6f66e56047785 P 9e19305103bc47a787612cf54d696491 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:37.902813 29399 raft_consensus.cc:504] T c8d21aa5e17345d685f6f66e56047785 P 9e19305103bc47a787612cf54d696491 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f43544884dd7407fb8265a8dd63fb9f3" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33465 } } peers { permanent_uuid: "9e19305103bc47a787612cf54d696491" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 41723 } } peers { permanent_uuid: "91889189682a4986a022a0a72d373c1b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 35933 } }
I20260806 02:49:37.903148 29399 leader_election.cc:288] T c8d21aa5e17345d685f6f66e56047785 P 9e19305103bc47a787612cf54d696491 [CANDIDATE]: Term 1 election: Requested vote from peers f43544884dd7407fb8265a8dd63fb9f3 (127.1.183.130:33465), 91889189682a4986a022a0a72d373c1b (127.1.183.129:35933)
I20260806 02:49:37.903519 28927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d21aa5e17345d685f6f66e56047785" candidate_uuid: "9e19305103bc47a787612cf54d696491" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91889189682a4986a022a0a72d373c1b"
I20260806 02:49:37.903522 29115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8d21aa5e17345d685f6f66e56047785" candidate_uuid: "9e19305103bc47a787612cf54d696491" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f43544884dd7407fb8265a8dd63fb9f3"
I20260806 02:49:37.903621 28927 raft_consensus.cc:3037] T c8d21aa5e17345d685f6f66e56047785 P 91889189682a4986a022a0a72d373c1b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:37.903648 29115 raft_consensus.cc:3037] T c8d21aa5e17345d685f6f66e56047785 P f43544884dd7407fb8265a8dd63fb9f3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:37.904755 28927 raft_consensus.cc:2454] T c8d21aa5e17345d685f6f66e56047785 P 91889189682a4986a022a0a72d373c1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e19305103bc47a787612cf54d696491 in term 1.
I20260806 02:49:37.904800 29115 raft_consensus.cc:2454] T c8d21aa5e17345d685f6f66e56047785 P f43544884dd7407fb8265a8dd63fb9f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e19305103bc47a787612cf54d696491 in term 1.
I20260806 02:49:37.905048 29208 leader_election.cc:302] T c8d21aa5e17345d685f6f66e56047785 P 9e19305103bc47a787612cf54d696491 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91889189682a4986a022a0a72d373c1b, 9e19305103bc47a787612cf54d696491; no voters:
I20260806 02:49:37.905256 29399 raft_consensus.cc:2781] T c8d21aa5e17345d685f6f66e56047785 P 9e19305103bc47a787612cf54d696491 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:49:37.905512 29399 raft_consensus.cc:686] T c8d21aa5e17345d685f6f66e56047785 P 9e19305103bc47a787612cf54d696491 [term 1 LEADER]: Becoming Leader. State: Replica: 9e19305103bc47a787612cf54d696491, State: Running, Role: LEADER
I20260806 02:49:37.905861 29399 consensus_queue.cc:237] T c8d21aa5e17345d685f6f66e56047785 P 9e19305103bc47a787612cf54d696491 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f43544884dd7407fb8265a8dd63fb9f3" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33465 } } peers { permanent_uuid: "9e19305103bc47a787612cf54d696491" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 41723 } } peers { permanent_uuid: "91889189682a4986a022a0a72d373c1b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 35933 } }
I20260806 02:49:37.909137 28718 catalog_manager.cc:5495] T c8d21aa5e17345d685f6f66e56047785 P 9e19305103bc47a787612cf54d696491 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e19305103bc47a787612cf54d696491 (127.1.183.131). New cstate: current_term: 1 leader_uuid: "9e19305103bc47a787612cf54d696491" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f43544884dd7407fb8265a8dd63fb9f3" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e19305103bc47a787612cf54d696491" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 41723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91889189682a4986a022a0a72d373c1b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 35933 } health_report { overall_health: UNKNOWN } } }
I20260806 02:49:37.921728 29385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:37.918100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45158 (local address 127.1.183.131:41723)
0806 02:49:37.918231 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:37.918238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:37.918253 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:49:37.918458 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:37.918461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:37.918512 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:37.918619 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:49:37.918863 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.919371 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.920275 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:37.920488 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:37.920599 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:37.920926 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:49:37.921305 (+ 379us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:49:37.921469 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:37.921558 (+ 89us) server_negotiation.cc:299] Negotiation successful
0806 02:49:37.921597 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:49:37.927644 29115 raft_consensus.cc:1261] T c8d21aa5e17345d685f6f66e56047785 P f43544884dd7407fb8265a8dd63fb9f3 [term 1 FOLLOWER]: Refusing update from remote peer 9e19305103bc47a787612cf54d696491: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:49:37.927641 28927 raft_consensus.cc:1261] T c8d21aa5e17345d685f6f66e56047785 P 91889189682a4986a022a0a72d373c1b [term 1 FOLLOWER]: Refusing update from remote peer 9e19305103bc47a787612cf54d696491: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:49:37.928418 29399 consensus_queue.cc:1035] T c8d21aa5e17345d685f6f66e56047785 P 9e19305103bc47a787612cf54d696491 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f43544884dd7407fb8265a8dd63fb9f3" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 33465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:49:37.928671 29401 consensus_queue.cc:1035] T c8d21aa5e17345d685f6f66e56047785 P 9e19305103bc47a787612cf54d696491 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91889189682a4986a022a0a72d373c1b" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 35933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:49:37.935401 29405 mvcc.cc:205] Tried to move back new op lower bound from 7315392831184113664 to 7315392831105449984. Current Snapshot: MvccSnapshot[applied={T|T < 7315392831184113664}]
I20260806 02:49:37.939250 29406 mvcc.cc:205] Tried to move back new op lower bound from 7315392831184113664 to 7315392831105449984. Current Snapshot: MvccSnapshot[applied={T|T < 7315392831184113664}]
I20260806 02:49:37.945582 28718 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:49:37.945732 28718 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:49:37.948035 28718 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e66769bf6984ec2a382f284d8552228: Sending DeleteTablet for 3 replicas of tablet c8d21aa5e17345d685f6f66e56047785
I20260806 02:49:37.949013 29275 tablet_service.cc:1514] Processing DeleteTablet for tablet c8d21aa5e17345d685f6f66e56047785 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:49:37 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:45148
I20260806 02:49:37.949020 29089 tablet_service.cc:1514] Processing DeleteTablet for tablet c8d21aa5e17345d685f6f66e56047785 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:49:37 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:48770
I20260806 02:49:37.949504 29426 tablet_replica.cc:330] T c8d21aa5e17345d685f6f66e56047785 P 9e19305103bc47a787612cf54d696491: stopping tablet replica
I20260806 02:49:37.949668 28898 tablet_service.cc:1514] Processing DeleteTablet for tablet c8d21aa5e17345d685f6f66e56047785 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:49:37 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:56380
I20260806 02:49:37.949781 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 28810
I20260806 02:49:37.949793 29426 raft_consensus.cc:2229] T c8d21aa5e17345d685f6f66e56047785 P 9e19305103bc47a787612cf54d696491 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:49:37.950177 29426 raft_consensus.cc:2258] T c8d21aa5e17345d685f6f66e56047785 P 9e19305103bc47a787612cf54d696491 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:49:37.952559 29429 tablet_replica.cc:330] T c8d21aa5e17345d685f6f66e56047785 P f43544884dd7407fb8265a8dd63fb9f3: stopping tablet replica
I20260806 02:49:37.954414 29429 raft_consensus.cc:2229] T c8d21aa5e17345d685f6f66e56047785 P f43544884dd7407fb8265a8dd63fb9f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:49:37.954808 29429 raft_consensus.cc:2258] T c8d21aa5e17345d685f6f66e56047785 P f43544884dd7407fb8265a8dd63fb9f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:49:37.957041 29429 ts_tablet_manager.cc:1905] T c8d21aa5e17345d685f6f66e56047785 P f43544884dd7407fb8265a8dd63fb9f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:49:37.959825 29426 ts_tablet_manager.cc:1905] T c8d21aa5e17345d685f6f66e56047785 P 9e19305103bc47a787612cf54d696491: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:49:37.960026 29429 ts_tablet_manager.cc:1918] T c8d21aa5e17345d685f6f66e56047785 P f43544884dd7407fb8265a8dd63fb9f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:49:37.960227 29429 log.cc:1196] T c8d21aa5e17345d685f6f66e56047785 P f43544884dd7407fb8265a8dd63fb9f3: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-1/wal/wals/c8d21aa5e17345d685f6f66e56047785
I20260806 02:49:37.960671 29429 ts_tablet_manager.cc:1939] T c8d21aa5e17345d685f6f66e56047785 P f43544884dd7407fb8265a8dd63fb9f3: Deleting consensus metadata
I20260806 02:49:37.962059 28701 catalog_manager.cc:4841] TS f43544884dd7407fb8265a8dd63fb9f3 (127.1.183.130:33465): tablet c8d21aa5e17345d685f6f66e56047785 (table test-table [id=c1413e51fdf242d0b4de876e8bcb737b]) successfully deleted
I20260806 02:49:37.963305 29426 ts_tablet_manager.cc:1918] T c8d21aa5e17345d685f6f66e56047785 P 9e19305103bc47a787612cf54d696491: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:49:37.963424 29426 log.cc:1196] T c8d21aa5e17345d685f6f66e56047785 P 9e19305103bc47a787612cf54d696491: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984313887056-1758-0/minicluster-data/ts-2/wal/wals/c8d21aa5e17345d685f6f66e56047785
I20260806 02:49:37.963835 29426 ts_tablet_manager.cc:1939] T c8d21aa5e17345d685f6f66e56047785 P 9e19305103bc47a787612cf54d696491: Deleting consensus metadata
I20260806 02:49:37.965121 28701 catalog_manager.cc:4841] TS 9e19305103bc47a787612cf54d696491 (127.1.183.131:41723): tablet c8d21aa5e17345d685f6f66e56047785 (table test-table [id=c1413e51fdf242d0b4de876e8bcb737b]) successfully deleted
W20260806 02:49:37.965677 28701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.183.129:35933 (error 108)
I20260806 02:49:37.965945 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 28998
I20260806 02:49:37.972427 29384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:37.972076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:35933 (local address 127.0.0.1:56396)
0806 02:49:37.972237 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:37.972319 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.129:35933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 02:49:37.972668 28701 catalog_manager.cc:4584] TS 91889189682a4986a022a0a72d373c1b (127.1.183.129:35933): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c8d21aa5e17345d685f6f66e56047785: Network error: Client connection negotiation failed: client connection to 127.1.183.129:35933: connect: Connection refused (error 111)
I20260806 02:49:37.981009 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 29185
I20260806 02:49:37.992048 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 28671
2026-08-06T02:49:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2489 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-06T02:49:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:49:38Z Disabled control of system clock
I20260806 02:49:38.030228 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:44099
--webserver_interface=127.1.183.190
--webserver_port=0
--builtin_ntp_servers=127.1.183.148:33325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:44099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:49:38.145892 29435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:38.146153 29435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:38.146189 29435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:38.150113 29435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:49:38.150174 29435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:38.150190 29435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:38.150207 29435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:49:38.150224 29435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:49:38.154911 29435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:33325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:44099
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:44099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:38.156070 29435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:38.157490 29435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:38.166370 29443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:38.166389 29445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:38.166370 29442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:38.166841 29435 server_base.cc:1034] running on GCE node
I20260806 02:49:38.167630 29435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:38.168879 29435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:38.170076 29435 hybrid_clock.cc:648] HybridClock initialized: now 1785984578170062 us; error 25 us; skew 500 ppm
I20260806 02:49:38.171972 29435 webserver.cc:460] Webserver started at http://127.1.183.190:36593/ using document root <none> and password file <none>
I20260806 02:49:38.172649 29435 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:38.172749 29435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:38.173044 29435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:38.175453 29435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "79f0ff26aa564005b391dcbed4ea04b1"
format_stamp: "Formatted at 2026-08-06 02:49:38 on dist-test-slave-gmjp"
server_key: "572d91543cd1e658f5cd2cbc7ab827da"
server_key_iv: "a471086128253873522f6addc49aa46c"
server_key_version: "encryptionkey@0"
I20260806 02:49:38.176057 29435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "79f0ff26aa564005b391dcbed4ea04b1"
format_stamp: "Formatted at 2026-08-06 02:49:38 on dist-test-slave-gmjp"
server_key: "572d91543cd1e658f5cd2cbc7ab827da"
server_key_iv: "a471086128253873522f6addc49aa46c"
server_key_version: "encryptionkey@0"
I20260806 02:49:38.179858 29435 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:49:38.182494 29452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:38.183764 29435 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 02:49:38.183917 29435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "79f0ff26aa564005b391dcbed4ea04b1"
format_stamp: "Formatted at 2026-08-06 02:49:38 on dist-test-slave-gmjp"
server_key: "572d91543cd1e658f5cd2cbc7ab827da"
server_key_iv: "a471086128253873522f6addc49aa46c"
server_key_version: "encryptionkey@0"
I20260806 02:49:38.184048 29435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:38.203814 29435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:38.204630 29435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:38.204849 29435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:38.211866 29435 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:44099
I20260806 02:49:38.211879 29521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:44099 every 8 connection(s)
I20260806 02:49:38.212906 29435 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:49:38.216274 29523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:38.216868 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 29435
I20260806 02:49:38.217010 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:49:38.217341 1758 external_mini_cluster.cc:1428] Setting key 7d07bb7e16fbcc72dfe7069650920df0
I20260806 02:49:38.224723 29523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79f0ff26aa564005b391dcbed4ea04b1: Bootstrap starting.
I20260806 02:49:38.227569 29523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79f0ff26aa564005b391dcbed4ea04b1: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:38.228649 29523 log.cc:824] T 00000000000000000000000000000000 P 79f0ff26aa564005b391dcbed4ea04b1: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:38.230858 29527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:38.218954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34628 (local address 127.1.183.190:44099)
0806 02:49:38.220745 (+ 1791us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:38.220769 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:38.220820 (+ 51us) server_negotiation.cc:407] Connection header received
0806 02:49:38.221910 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:38.221939 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:38.222527 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:38.222933 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:38.223477 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:38.225190 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:38.226504 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:38.227314 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:38.227506 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:38.228592 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:38.228601 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:38.228621 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:49:38.229185 (+ 564us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:38.229291 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:49:38.229370 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:38.229485 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:38.229902 (+ 417us) server_negotiation.cc:299] Negotiation successful
0806 02:49:38.230143 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1287,"thread_start_us":140,"threads_started":1}
I20260806 02:49:38.231251 29523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79f0ff26aa564005b391dcbed4ea04b1: No bootstrap required, opened a new log
I20260806 02:49:38.235971 29523 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79f0ff26aa564005b391dcbed4ea04b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79f0ff26aa564005b391dcbed4ea04b1" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 44099 } }
I20260806 02:49:38.236408 29523 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79f0ff26aa564005b391dcbed4ea04b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:38.236527 29523 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79f0ff26aa564005b391dcbed4ea04b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79f0ff26aa564005b391dcbed4ea04b1, State: Initialized, Role: FOLLOWER
I20260806 02:49:38.237079 29523 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79f0ff26aa564005b391dcbed4ea04b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79f0ff26aa564005b391dcbed4ea04b1" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 44099 } }
I20260806 02:49:38.237317 29523 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79f0ff26aa564005b391dcbed4ea04b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:49:38.237435 29523 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79f0ff26aa564005b391dcbed4ea04b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:49:38.237581 29523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79f0ff26aa564005b391dcbed4ea04b1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:38.238876 29523 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79f0ff26aa564005b391dcbed4ea04b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79f0ff26aa564005b391dcbed4ea04b1" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 44099 } }
I20260806 02:49:38.239336 29523 leader_election.cc:302] T 00000000000000000000000000000000 P 79f0ff26aa564005b391dcbed4ea04b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79f0ff26aa564005b391dcbed4ea04b1; no voters:
I20260806 02:49:38.239781 29523 leader_election.cc:288] T 00000000000000000000000000000000 P 79f0ff26aa564005b391dcbed4ea04b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:49:38.239903 29529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79f0ff26aa564005b391dcbed4ea04b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:49:38.240142 29529 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79f0ff26aa564005b391dcbed4ea04b1 [term 1 LEADER]: Becoming Leader. State: Replica: 79f0ff26aa564005b391dcbed4ea04b1, State: Running, Role: LEADER
I20260806 02:49:38.240517 29529 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79f0ff26aa564005b391dcbed4ea04b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79f0ff26aa564005b391dcbed4ea04b1" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 44099 } }
I20260806 02:49:38.241403 29523 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79f0ff26aa564005b391dcbed4ea04b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:49:38.241772 29529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79f0ff26aa564005b391dcbed4ea04b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79f0ff26aa564005b391dcbed4ea04b1. Latest consensus state: current_term: 1 leader_uuid: "79f0ff26aa564005b391dcbed4ea04b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79f0ff26aa564005b391dcbed4ea04b1" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 44099 } } }
I20260806 02:49:38.241914 29529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79f0ff26aa564005b391dcbed4ea04b1 [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:38.243883 29536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:49:38.244248 29531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79f0ff26aa564005b391dcbed4ea04b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79f0ff26aa564005b391dcbed4ea04b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79f0ff26aa564005b391dcbed4ea04b1" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 44099 } } }
I20260806 02:49:38.244436 29531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79f0ff26aa564005b391dcbed4ea04b1 [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:38.247412 29536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:49:38.253772 29536 catalog_manager.cc:1347] Generated new cluster ID: 468a420dfb30458495e2505f019958e3
I20260806 02:49:38.253835 29536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:49:38.263844 29536 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:49:38.264806 29536 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:49:38.275070 29536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79f0ff26aa564005b391dcbed4ea04b1: Generated new TSK 0
I20260806 02:49:38.275905 29536 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:49:38.288560 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.129:0
--local_ip_for_outbound_sockets=127.1.183.129
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:44099
--builtin_ntp_servers=127.1.183.148:33325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:49:38.415966 29557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:38.416334 29557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:38.416416 29557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:38.420655 29557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:38.420725 29557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:38.420817 29557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.129
I20260806 02:49:38.425772 29557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:33325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:44099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.129
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:38.426923 29557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:38.428424 29557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:38.438551 29564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:38.438721 29557 server_base.cc:1034] running on GCE node
W20260806 02:49:38.438552 29566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:38.439050 29569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:38.439478 29557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:38.440235 29557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:38.441423 29557 hybrid_clock.cc:648] HybridClock initialized: now 1785984578441390 us; error 53 us; skew 500 ppm
I20260806 02:49:38.443636 29557 webserver.cc:460] Webserver started at http://127.1.183.129:45533/ using document root <none> and password file <none>
I20260806 02:49:38.444388 29557 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:38.444459 29557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:38.444695 29557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:38.446842 29557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-0/data/instance:
uuid: "251ddbe523db4a46a9fec795d140e75e"
format_stamp: "Formatted at 2026-08-06 02:49:38 on dist-test-slave-gmjp"
server_key: "f6e68b20eb7f1cda98ec031a739512d3"
server_key_iv: "e5f3253d3ff73c8a193c882e4e839626"
server_key_version: "encryptionkey@0"
I20260806 02:49:38.447449 29557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance:
uuid: "251ddbe523db4a46a9fec795d140e75e"
format_stamp: "Formatted at 2026-08-06 02:49:38 on dist-test-slave-gmjp"
server_key: "f6e68b20eb7f1cda98ec031a739512d3"
server_key_iv: "e5f3253d3ff73c8a193c882e4e839626"
server_key_version: "encryptionkey@0"
I20260806 02:49:38.451633 29557 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:49:38.454312 29582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:38.455600 29557 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:49:38.455739 29557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-0/wal
uuid: "251ddbe523db4a46a9fec795d140e75e"
format_stamp: "Formatted at 2026-08-06 02:49:38 on dist-test-slave-gmjp"
server_key: "f6e68b20eb7f1cda98ec031a739512d3"
server_key_iv: "e5f3253d3ff73c8a193c882e4e839626"
server_key_version: "encryptionkey@0"
I20260806 02:49:38.455858 29557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:38.469926 29557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:38.470597 29557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:38.470798 29557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:38.471383 29557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:49:38.472409 29557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:49:38.472483 29557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:38.472558 29557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:49:38.472600 29557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:38.481335 29557 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.129:40751
I20260806 02:49:38.481374 29739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.129:40751 every 8 connection(s)
I20260806 02:49:38.482245 29557 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
I20260806 02:49:38.486220 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 29557
I20260806 02:49:38.486363 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance
I20260806 02:49:38.486660 1758 external_mini_cluster.cc:1428] Setting key dccca10ac15536f0b2c6293059bf38f9
I20260806 02:49:38.488986 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.130:0
--local_ip_for_outbound_sockets=127.1.183.130
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:44099
--builtin_ntp_servers=127.1.183.148:33325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:49:38.492949 29527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:38.484259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:53559 (local address 127.1.183.190:44099)
0806 02:49:38.484435 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:38.484440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:38.485511 (+ 1071us) server_negotiation.cc:407] Connection header received
0806 02:49:38.487013 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:38.487017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:38.487079 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:38.487215 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:38.488574 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:38.489382 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:38.490531 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:38.490899 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:38.490986 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:38.492396 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:38.492400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:38.492402 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:49:38.492422 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:38.492441 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:49:38.492449 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:38.492531 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:38.492721 (+ 190us) server_negotiation.cc:299] Negotiation successful
0806 02:49:38.492788 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 02:49:38.493824 29744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:38.484685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:44099 (local address 127.1.183.129:53559)
0806 02:49:38.485347 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:38.485390 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:38.486810 (+ 1420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:38.487473 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:38.487498 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:38.487605 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:49:38.488394 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:38.488422 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:38.489517 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:38.489519 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:38.490404 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:38.490413 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:38.491075 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:38.491082 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:38.491310 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:38.492036 (+ 726us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:49:38.492038 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:49:38.492095 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:49:38.492225 (+ 130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:49:38.492533 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:49:38.492546 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:49:38.492554 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 02:49:38.492698 (+ 144us) client_negotiation.cc:240] Negotiation successful
0806 02:49:38.492992 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":115,"threads_started":1}
I20260806 02:49:38.495288 29740 heartbeater.cc:347] Connected to a master server at 127.1.183.190:44099
I20260806 02:49:38.495548 29740 heartbeater.cc:464] Registering TS with master...
I20260806 02:49:38.496119 29740 heartbeater.cc:511] Master 127.1.183.190:44099 requested a full tablet report, sending...
I20260806 02:49:38.497973 29478 ts_manager.cc:194] Registered new tserver with Master: 251ddbe523db4a46a9fec795d140e75e (127.1.183.129:40751)
I20260806 02:49:38.499787 29478 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.183.129:53559
W20260806 02:49:38.607334 29745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:38.607605 29745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:38.607645 29745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:38.611586 29745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:38.611640 29745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:38.611693 29745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.130
I20260806 02:49:38.616518 29745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:33325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:44099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.130
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:38.617558 29745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:38.618939 29745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:38.628717 29751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:38.628849 29745 server_base.cc:1034] running on GCE node
W20260806 02:49:38.628786 29756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:38.628763 29758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:38.629463 29745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:38.630110 29745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:38.631302 29745 hybrid_clock.cc:648] HybridClock initialized: now 1785984578631277 us; error 34 us; skew 500 ppm
I20260806 02:49:38.633426 29745 webserver.cc:460] Webserver started at http://127.1.183.130:41163/ using document root <none> and password file <none>
I20260806 02:49:38.634053 29745 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:38.634119 29745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:38.634385 29745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:38.636602 29745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-1/data/instance:
uuid: "3fecd462e89b46a7920aa054e1675924"
format_stamp: "Formatted at 2026-08-06 02:49:38 on dist-test-slave-gmjp"
server_key: "bcb0a63cb654c5b892d89b28182698a3"
server_key_iv: "82082641c64535f51cd9d8e38d34e0d1"
server_key_version: "encryptionkey@0"
I20260806 02:49:38.637221 29745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance:
uuid: "3fecd462e89b46a7920aa054e1675924"
format_stamp: "Formatted at 2026-08-06 02:49:38 on dist-test-slave-gmjp"
server_key: "bcb0a63cb654c5b892d89b28182698a3"
server_key_iv: "82082641c64535f51cd9d8e38d34e0d1"
server_key_version: "encryptionkey@0"
I20260806 02:49:38.641103 29745 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260806 02:49:38.643784 29767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:38.645080 29745 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260806 02:49:38.645180 29745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-1/wal
uuid: "3fecd462e89b46a7920aa054e1675924"
format_stamp: "Formatted at 2026-08-06 02:49:38 on dist-test-slave-gmjp"
server_key: "bcb0a63cb654c5b892d89b28182698a3"
server_key_iv: "82082641c64535f51cd9d8e38d34e0d1"
server_key_version: "encryptionkey@0"
I20260806 02:49:38.645272 29745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:38.669433 29745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:38.670135 29745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:38.670296 29745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:38.670852 29745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:49:38.671806 29745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:49:38.671855 29745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:38.671904 29745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:49:38.671921 29745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:38.681720 29745 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.130:42605
I20260806 02:49:38.681741 29932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.130:42605 every 8 connection(s)
I20260806 02:49:38.682710 29745 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
I20260806 02:49:38.687263 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 29745
I20260806 02:49:38.687420 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance
I20260806 02:49:38.687810 1758 external_mini_cluster.cc:1428] Setting key 969a8c169c7eef92b8f2b102320cb289
I20260806 02:49:38.690083 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.131:0
--local_ip_for_outbound_sockets=127.1.183.131
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:44099
--builtin_ntp_servers=127.1.183.148:33325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:49:38.693552 29527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:38.684671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:46857 (local address 127.1.183.190:44099)
0806 02:49:38.684820 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:38.684824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:38.685897 (+ 1073us) server_negotiation.cc:407] Connection header received
0806 02:49:38.687552 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:38.687557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:38.687622 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:38.687710 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:38.688882 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:38.689514 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:38.691233 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:38.691443 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:38.691567 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:38.692980 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:38.692983 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:38.692986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:49:38.692997 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:38.693018 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:49:38.693026 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:38.693091 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:38.693356 (+ 265us) server_negotiation.cc:299] Negotiation successful
0806 02:49:38.693437 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:49:38.694736 29935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:38.685083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:44099 (local address 127.1.183.130:46857)
0806 02:49:38.685748 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:38.685788 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:38.687229 (+ 1441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:38.687951 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:38.687961 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:38.688074 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:49:38.688744 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:38.688764 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:38.689646 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:38.689649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:38.691108 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:38.691124 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:38.691599 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:38.691606 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:38.691841 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:38.692649 (+ 808us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:49:38.692654 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:49:38.692713 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:49:38.692819 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:49:38.693116 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:49:38.693127 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:49:38.693134 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 02:49:38.693334 (+ 200us) client_negotiation.cc:240] Negotiation successful
0806 02:49:38.693671 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":117,"threads_started":1}
I20260806 02:49:38.696197 29933 heartbeater.cc:347] Connected to a master server at 127.1.183.190:44099
I20260806 02:49:38.696465 29933 heartbeater.cc:464] Registering TS with master...
I20260806 02:49:38.697017 29933 heartbeater.cc:511] Master 127.1.183.190:44099 requested a full tablet report, sending...
I20260806 02:49:38.698136 29478 ts_manager.cc:194] Registered new tserver with Master: 3fecd462e89b46a7920aa054e1675924 (127.1.183.130:42605)
I20260806 02:49:38.698726 29478 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.183.130:46857
W20260806 02:49:38.805816 29938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:38.806131 29938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:38.806201 29938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:38.810302 29938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:38.810395 29938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:38.810489 29938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.131
I20260806 02:49:38.815351 29938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:33325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:44099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.131
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:38.816601 29938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:38.818120 29938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:38.827768 29944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:38.827965 29945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:38.828016 29938 server_base.cc:1034] running on GCE node
W20260806 02:49:38.828209 29948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:38.828662 29938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:38.829315 29938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:38.830513 29938 hybrid_clock.cc:648] HybridClock initialized: now 1785984578830480 us; error 45 us; skew 500 ppm
I20260806 02:49:38.832429 29938 webserver.cc:460] Webserver started at http://127.1.183.131:36281/ using document root <none> and password file <none>
I20260806 02:49:38.833029 29938 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:38.833124 29938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:38.833366 29938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:38.835431 29938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-2/data/instance:
uuid: "e8f0c9ad2e9b47d781693cea10cf0c83"
format_stamp: "Formatted at 2026-08-06 02:49:38 on dist-test-slave-gmjp"
server_key: "d88d9d8208084681b3844870cc65e480"
server_key_iv: "b876497f05d7471fb68f9af65a655a3c"
server_key_version: "encryptionkey@0"
I20260806 02:49:38.836018 29938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance:
uuid: "e8f0c9ad2e9b47d781693cea10cf0c83"
format_stamp: "Formatted at 2026-08-06 02:49:38 on dist-test-slave-gmjp"
server_key: "d88d9d8208084681b3844870cc65e480"
server_key_iv: "b876497f05d7471fb68f9af65a655a3c"
server_key_version: "encryptionkey@0"
I20260806 02:49:38.839731 29938 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260806 02:49:38.842422 29957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:38.843683 29938 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:49:38.843905 29938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-2/wal
uuid: "e8f0c9ad2e9b47d781693cea10cf0c83"
format_stamp: "Formatted at 2026-08-06 02:49:38 on dist-test-slave-gmjp"
server_key: "d88d9d8208084681b3844870cc65e480"
server_key_iv: "b876497f05d7471fb68f9af65a655a3c"
server_key_version: "encryptionkey@0"
I20260806 02:49:38.844012 29938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:38.856756 29938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:38.857483 29938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:38.857681 29938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:38.858302 29938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:49:38.859354 29938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:49:38.859429 29938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:38.859498 29938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:49:38.859537 29938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:38.868511 29938 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.131:46593
I20260806 02:49:38.868553 30135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.131:46593 every 8 connection(s)
I20260806 02:49:38.869484 29938 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
I20260806 02:49:38.878538 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 29938
I20260806 02:49:38.878670 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance
I20260806 02:49:38.878974 1758 external_mini_cluster.cc:1428] Setting key f2a7b7a822226cab99ae625ae64fceaa
I20260806 02:49:38.879285 29527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:38.871433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:33491 (local address 127.1.183.190:44099)
0806 02:49:38.871581 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:38.871585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:38.872689 (+ 1104us) server_negotiation.cc:407] Connection header received
0806 02:49:38.874166 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:38.874169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:38.874231 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:38.874341 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:38.875407 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:38.875946 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:38.877208 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:38.877442 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:38.877559 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:38.878753 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:38.878756 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:38.878758 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:49:38.878771 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:38.878791 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:49:38.878798 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:38.878897 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:38.879071 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 02:49:38.879135 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:49:38.880273 30142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:38.871776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:44099 (local address 127.1.183.131:33491)
0806 02:49:38.872529 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:38.872571 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:38.873935 (+ 1364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:38.874550 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:38.874557 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:38.874643 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:49:38.875245 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:38.875260 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:38.876094 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:38.876096 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:38.877078 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:38.877089 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:38.877608 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:38.877612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:38.877783 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:38.878446 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:49:38.878461 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:49:38.878502 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:49:38.878571 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:49:38.878902 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:49:38.878915 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:49:38.878923 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 02:49:38.879065 (+ 142us) client_negotiation.cc:240] Negotiation successful
0806 02:49:38.879358 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":120,"threads_started":1}
I20260806 02:49:38.881517 30136 heartbeater.cc:347] Connected to a master server at 127.1.183.190:44099
I20260806 02:49:38.881771 30136 heartbeater.cc:464] Registering TS with master...
I20260806 02:49:38.882302 30136 heartbeater.cc:511] Master 127.1.183.190:44099 requested a full tablet report, sending...
I20260806 02:49:38.883302 29478 ts_manager.cc:194] Registered new tserver with Master: e8f0c9ad2e9b47d781693cea10cf0c83 (127.1.183.131:46593)
I20260806 02:49:38.883868 29478 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.183.131:33491
I20260806 02:49:38.892937 1758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:49:38.898947 29527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:38.894932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34634 (local address 127.1.183.190:44099)
0806 02:49:38.895108 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:38.895113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:38.895256 (+ 143us) server_negotiation.cc:407] Connection header received
0806 02:49:38.895422 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:38.895426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:38.895470 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:38.895553 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:38.895854 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:38.896417 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:38.897261 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:38.897476 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:38.897579 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:38.898441 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:38.898445 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:38.898460 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:49:38.898472 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:38.898490 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:49:38.898497 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:38.898632 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:38.898761 (+ 129us) server_negotiation.cc:299] Negotiation successful
0806 02:49:38.898800 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 02:49:38.901873 29478 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34634:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:49:38.904914 29478 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:49:38.919763 30157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:38.914184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34668 (local address 127.1.183.129:40751)
0806 02:49:38.914736 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:38.914749 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:38.914774 (+ 25us) server_negotiation.cc:407] Connection header received
0806 02:49:38.914855 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:38.914860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:38.915055 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:38.915531 (+ 476us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:38.915671 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:38.916784 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:38.918540 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:38.919062 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:38.919190 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:38.919390 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:38.919479 (+ 89us) server_negotiation.cc:299] Negotiation successful
0806 02:49:38.919590 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":103,"threads_started":1}
I20260806 02:49:38.919759 30152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:38.913539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:40751 (local address 127.0.0.1:34668)
0806 02:49:38.914185 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:38.914199 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:38.914275 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:38.915217 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:38.915222 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:38.915243 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:38.915368 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:38.915376 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:38.916912 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:38.916915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:38.918366 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:38.918376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:38.919189 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:38.919193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:38.919297 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:38.919334 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 02:49:38.919453 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 02:49:38.919597 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":145,"thread_start_us":111,"threads_started":1}
I20260806 02:49:38.919770 30154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:38.913466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:46593 (local address 127.0.0.1:34536)
0806 02:49:38.913937 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:38.913984 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:38.914119 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:38.915071 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:38.915077 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:38.915101 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:38.915256 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:38.915265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:38.916781 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:38.916784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:38.918154 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:38.918164 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:38.919253 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:38.919258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:38.919354 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:38.919376 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:49:38.919428 (+ 52us) client_negotiation.cc:240] Negotiation successful
0806 02:49:38.919592 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":78,"thread_start_us":96,"threads_started":1}
I20260806 02:49:38.920387 30156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:38.913571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34536 (local address 127.1.183.131:46593)
0806 02:49:38.914556 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:38.914570 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:38.914594 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:49:38.914683 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:38.914688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:38.914911 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:38.915081 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:38.915375 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:38.916659 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:38.918280 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:38.919103 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:38.919841 (+ 738us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:38.920043 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:38.920096 (+ 53us) server_negotiation.cc:299] Negotiation successful
0806 02:49:38.920225 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":876,"thread_start_us":167,"threads_started":1}
I20260806 02:49:38.920495 30160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:38.914507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48804 (local address 127.1.183.130:42605)
0806 02:49:38.914960 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:38.914976 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:38.915466 (+ 490us) server_negotiation.cc:407] Connection header received
0806 02:49:38.915592 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:38.915597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:38.915727 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:38.915881 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:38.916085 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:38.916968 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:38.918516 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:38.919253 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:38.919345 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:38.919546 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:38.920223 (+ 677us) server_negotiation.cc:299] Negotiation successful
0806 02:49:38.920319 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":148,"threads_started":1}
I20260806 02:49:38.920681 30159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:38.914268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:42605 (local address 127.0.0.1:48804)
0806 02:49:38.915336 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:38.915351 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:38.915475 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:38.915862 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:38.915865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:38.915883 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:38.915964 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:38.915969 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:38.917079 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:38.917082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:38.918365 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:38.918377 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:38.919607 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:38.919610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:38.919695 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:38.919715 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 02:49:38.919786 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 02:49:38.919852 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"mutex_wait_us":68,"thread_start_us":106,"threads_started":1}
I20260806 02:49:38.923372 29839 tablet_service.cc:1467] Processing CreateTablet for tablet 36205a0bae1c4d1f971d80305f5699fa (DEFAULT_TABLE table=test-table [id=f152629c387743ae8e19b7c338395337]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:49:38.923394 30032 tablet_service.cc:1467] Processing CreateTablet for tablet 36205a0bae1c4d1f971d80305f5699fa (DEFAULT_TABLE table=test-table [id=f152629c387743ae8e19b7c338395337]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:49:38.923396 29650 tablet_service.cc:1467] Processing CreateTablet for tablet 36205a0bae1c4d1f971d80305f5699fa (DEFAULT_TABLE table=test-table [id=f152629c387743ae8e19b7c338395337]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:49:38.924434 29839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36205a0bae1c4d1f971d80305f5699fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:38.924449 29650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36205a0bae1c4d1f971d80305f5699fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:38.924460 30032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36205a0bae1c4d1f971d80305f5699fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:38.930819 30161 tablet_bootstrap.cc:492] T 36205a0bae1c4d1f971d80305f5699fa P 3fecd462e89b46a7920aa054e1675924: Bootstrap starting.
I20260806 02:49:38.932377 30163 tablet_bootstrap.cc:492] T 36205a0bae1c4d1f971d80305f5699fa P e8f0c9ad2e9b47d781693cea10cf0c83: Bootstrap starting.
I20260806 02:49:38.932382 30164 tablet_bootstrap.cc:492] T 36205a0bae1c4d1f971d80305f5699fa P 251ddbe523db4a46a9fec795d140e75e: Bootstrap starting.
I20260806 02:49:38.933182 30161 tablet_bootstrap.cc:654] T 36205a0bae1c4d1f971d80305f5699fa P 3fecd462e89b46a7920aa054e1675924: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:38.934026 30161 log.cc:824] T 36205a0bae1c4d1f971d80305f5699fa P 3fecd462e89b46a7920aa054e1675924: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:38.934571 30163 tablet_bootstrap.cc:654] T 36205a0bae1c4d1f971d80305f5699fa P e8f0c9ad2e9b47d781693cea10cf0c83: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:38.934571 30164 tablet_bootstrap.cc:654] T 36205a0bae1c4d1f971d80305f5699fa P 251ddbe523db4a46a9fec795d140e75e: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:38.935413 30163 log.cc:824] T 36205a0bae1c4d1f971d80305f5699fa P e8f0c9ad2e9b47d781693cea10cf0c83: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:38.935413 30164 log.cc:824] T 36205a0bae1c4d1f971d80305f5699fa P 251ddbe523db4a46a9fec795d140e75e: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:38.937178 30161 tablet_bootstrap.cc:492] T 36205a0bae1c4d1f971d80305f5699fa P 3fecd462e89b46a7920aa054e1675924: No bootstrap required, opened a new log
I20260806 02:49:38.937194 30164 tablet_bootstrap.cc:492] T 36205a0bae1c4d1f971d80305f5699fa P 251ddbe523db4a46a9fec795d140e75e: No bootstrap required, opened a new log
I20260806 02:49:38.937399 30164 ts_tablet_manager.cc:1397] T 36205a0bae1c4d1f971d80305f5699fa P 251ddbe523db4a46a9fec795d140e75e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:49:38.937400 30161 ts_tablet_manager.cc:1397] T 36205a0bae1c4d1f971d80305f5699fa P 3fecd462e89b46a7920aa054e1675924: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 02:49:38.937821 30163 tablet_bootstrap.cc:492] T 36205a0bae1c4d1f971d80305f5699fa P e8f0c9ad2e9b47d781693cea10cf0c83: No bootstrap required, opened a new log
I20260806 02:49:38.938009 30163 ts_tablet_manager.cc:1397] T 36205a0bae1c4d1f971d80305f5699fa P e8f0c9ad2e9b47d781693cea10cf0c83: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:49:38.940610 30164 raft_consensus.cc:348] T 36205a0bae1c4d1f971d80305f5699fa P 251ddbe523db4a46a9fec795d140e75e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f0c9ad2e9b47d781693cea10cf0c83" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46593 } } peers { permanent_uuid: "251ddbe523db4a46a9fec795d140e75e" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40751 } } peers { permanent_uuid: "3fecd462e89b46a7920aa054e1675924" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 42605 } }
I20260806 02:49:38.940847 30164 raft_consensus.cc:374] T 36205a0bae1c4d1f971d80305f5699fa P 251ddbe523db4a46a9fec795d140e75e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:38.940932 30164 raft_consensus.cc:729] T 36205a0bae1c4d1f971d80305f5699fa P 251ddbe523db4a46a9fec795d140e75e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 251ddbe523db4a46a9fec795d140e75e, State: Initialized, Role: FOLLOWER
I20260806 02:49:38.941426 30164 consensus_queue.cc:260] T 36205a0bae1c4d1f971d80305f5699fa P 251ddbe523db4a46a9fec795d140e75e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f0c9ad2e9b47d781693cea10cf0c83" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46593 } } peers { permanent_uuid: "251ddbe523db4a46a9fec795d140e75e" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40751 } } peers { permanent_uuid: "3fecd462e89b46a7920aa054e1675924" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 42605 } }
I20260806 02:49:38.941473 30161 raft_consensus.cc:348] T 36205a0bae1c4d1f971d80305f5699fa P 3fecd462e89b46a7920aa054e1675924 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f0c9ad2e9b47d781693cea10cf0c83" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46593 } } peers { permanent_uuid: "251ddbe523db4a46a9fec795d140e75e" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40751 } } peers { permanent_uuid: "3fecd462e89b46a7920aa054e1675924" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 42605 } }
I20260806 02:49:38.941737 30161 raft_consensus.cc:374] T 36205a0bae1c4d1f971d80305f5699fa P 3fecd462e89b46a7920aa054e1675924 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:38.941859 30161 raft_consensus.cc:729] T 36205a0bae1c4d1f971d80305f5699fa P 3fecd462e89b46a7920aa054e1675924 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fecd462e89b46a7920aa054e1675924, State: Initialized, Role: FOLLOWER
I20260806 02:49:38.942109 29740 heartbeater.cc:503] Master 127.1.183.190:44099 was elected leader, sending a full tablet report...
I20260806 02:49:38.942469 30164 ts_tablet_manager.cc:1428] T 36205a0bae1c4d1f971d80305f5699fa P 251ddbe523db4a46a9fec795d140e75e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 02:49:38.942633 30163 raft_consensus.cc:348] T 36205a0bae1c4d1f971d80305f5699fa P e8f0c9ad2e9b47d781693cea10cf0c83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f0c9ad2e9b47d781693cea10cf0c83" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46593 } } peers { permanent_uuid: "251ddbe523db4a46a9fec795d140e75e" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40751 } } peers { permanent_uuid: "3fecd462e89b46a7920aa054e1675924" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 42605 } }
I20260806 02:49:38.942974 30163 raft_consensus.cc:374] T 36205a0bae1c4d1f971d80305f5699fa P e8f0c9ad2e9b47d781693cea10cf0c83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:38.943070 30163 raft_consensus.cc:729] T 36205a0bae1c4d1f971d80305f5699fa P e8f0c9ad2e9b47d781693cea10cf0c83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f0c9ad2e9b47d781693cea10cf0c83, State: Initialized, Role: FOLLOWER
I20260806 02:49:38.942996 30161 consensus_queue.cc:260] T 36205a0bae1c4d1f971d80305f5699fa P 3fecd462e89b46a7920aa054e1675924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f0c9ad2e9b47d781693cea10cf0c83" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46593 } } peers { permanent_uuid: "251ddbe523db4a46a9fec795d140e75e" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40751 } } peers { permanent_uuid: "3fecd462e89b46a7920aa054e1675924" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 42605 } }
I20260806 02:49:38.943531 30163 consensus_queue.cc:260] T 36205a0bae1c4d1f971d80305f5699fa P e8f0c9ad2e9b47d781693cea10cf0c83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f0c9ad2e9b47d781693cea10cf0c83" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46593 } } peers { permanent_uuid: "251ddbe523db4a46a9fec795d140e75e" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40751 } } peers { permanent_uuid: "3fecd462e89b46a7920aa054e1675924" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 42605 } }
I20260806 02:49:38.943701 29933 heartbeater.cc:503] Master 127.1.183.190:44099 was elected leader, sending a full tablet report...
I20260806 02:49:38.944023 30161 ts_tablet_manager.cc:1428] T 36205a0bae1c4d1f971d80305f5699fa P 3fecd462e89b46a7920aa054e1675924: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 02:49:38.944519 30163 ts_tablet_manager.cc:1428] T 36205a0bae1c4d1f971d80305f5699fa P e8f0c9ad2e9b47d781693cea10cf0c83: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 02:49:38.944610 30136 heartbeater.cc:503] Master 127.1.183.190:44099 was elected leader, sending a full tablet report...
W20260806 02:49:38.983525 29742 tablet.cc:2369] T 36205a0bae1c4d1f971d80305f5699fa P 251ddbe523db4a46a9fec795d140e75e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:49:39.012104 30173 raft_consensus.cc:482] T 36205a0bae1c4d1f971d80305f5699fa P e8f0c9ad2e9b47d781693cea10cf0c83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:49:39.012410 30173 raft_consensus.cc:504] T 36205a0bae1c4d1f971d80305f5699fa P e8f0c9ad2e9b47d781693cea10cf0c83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f0c9ad2e9b47d781693cea10cf0c83" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46593 } } peers { permanent_uuid: "251ddbe523db4a46a9fec795d140e75e" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40751 } } peers { permanent_uuid: "3fecd462e89b46a7920aa054e1675924" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 42605 } }
I20260806 02:49:39.013661 30173 leader_election.cc:288] T 36205a0bae1c4d1f971d80305f5699fa P e8f0c9ad2e9b47d781693cea10cf0c83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 251ddbe523db4a46a9fec795d140e75e (127.1.183.129:40751), 3fecd462e89b46a7920aa054e1675924 (127.1.183.130:42605)
I20260806 02:49:39.017589 30142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:39.013899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:40751 (local address 127.1.183.131:41413)
0806 02:49:39.014080 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:39.014097 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:39.014197 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:39.014546 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:39.014551 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:39.014572 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:39.014706 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:39.014714 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:39.015544 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:39.015547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:39.016412 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:39.016422 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:39.017121 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:39.017125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:39.017241 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:39.017279 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 02:49:39.017351 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 02:49:39.017445 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260806 02:49:39.017627 30157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:39.014007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:41413 (local address 127.1.183.129:40751)
0806 02:49:39.014145 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:39.014150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:39.014166 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:49:39.014340 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:39.014344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:39.014425 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:39.014508 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:39.014835 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:39.015423 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:39.016534 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:39.016997 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:39.017093 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:39.017133 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:39.017383 (+ 250us) server_negotiation.cc:299] Negotiation successful
0806 02:49:39.017472 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 02:49:39.017799 30175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:39.013924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:42605 (local address 127.1.183.131:60193)
0806 02:49:39.014359 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:39.014374 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:39.014483 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:39.014735 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:39.014738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:39.014753 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:39.014919 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:39.014926 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:39.015682 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:39.015684 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:39.016601 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:39.016612 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:39.017362 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:39.017365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:39.017462 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:39.017486 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 02:49:39.017566 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 02:49:39.017626 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":10,"thread_start_us":131,"threads_started":1}
I20260806 02:49:39.017830 30160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:39.014189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:60193 (local address 127.1.183.130:42605)
0806 02:49:39.014352 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:39.014356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:39.014458 (+ 102us) server_negotiation.cc:407] Connection header received
0806 02:49:39.014576 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:39.014579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:39.014627 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:39.014709 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:39.015043 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:39.015563 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:39.016734 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:39.017197 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:39.017314 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:39.017370 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:39.017601 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 02:49:39.017683 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:49:39.018505 29676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36205a0bae1c4d1f971d80305f5699fa" candidate_uuid: "e8f0c9ad2e9b47d781693cea10cf0c83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "251ddbe523db4a46a9fec795d140e75e" is_pre_election: true
I20260806 02:49:39.018513 29865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36205a0bae1c4d1f971d80305f5699fa" candidate_uuid: "e8f0c9ad2e9b47d781693cea10cf0c83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fecd462e89b46a7920aa054e1675924" is_pre_election: true
I20260806 02:49:39.018841 29676 raft_consensus.cc:2454] T 36205a0bae1c4d1f971d80305f5699fa P 251ddbe523db4a46a9fec795d140e75e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8f0c9ad2e9b47d781693cea10cf0c83 in term 0.
I20260806 02:49:39.018841 29865 raft_consensus.cc:2454] T 36205a0bae1c4d1f971d80305f5699fa P 3fecd462e89b46a7920aa054e1675924 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8f0c9ad2e9b47d781693cea10cf0c83 in term 0.
I20260806 02:49:39.019405 29959 leader_election.cc:302] T 36205a0bae1c4d1f971d80305f5699fa P e8f0c9ad2e9b47d781693cea10cf0c83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 251ddbe523db4a46a9fec795d140e75e, 3fecd462e89b46a7920aa054e1675924, e8f0c9ad2e9b47d781693cea10cf0c83; no voters:
I20260806 02:49:39.019685 30173 raft_consensus.cc:2781] T 36205a0bae1c4d1f971d80305f5699fa P e8f0c9ad2e9b47d781693cea10cf0c83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:49:39.019773 30173 raft_consensus.cc:482] T 36205a0bae1c4d1f971d80305f5699fa P e8f0c9ad2e9b47d781693cea10cf0c83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:49:39.019816 30173 raft_consensus.cc:3037] T 36205a0bae1c4d1f971d80305f5699fa P e8f0c9ad2e9b47d781693cea10cf0c83 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:39.020931 30173 raft_consensus.cc:504] T 36205a0bae1c4d1f971d80305f5699fa P e8f0c9ad2e9b47d781693cea10cf0c83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f0c9ad2e9b47d781693cea10cf0c83" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46593 } } peers { permanent_uuid: "251ddbe523db4a46a9fec795d140e75e" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40751 } } peers { permanent_uuid: "3fecd462e89b46a7920aa054e1675924" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 42605 } }
I20260806 02:49:39.021271 30173 leader_election.cc:288] T 36205a0bae1c4d1f971d80305f5699fa P e8f0c9ad2e9b47d781693cea10cf0c83 [CANDIDATE]: Term 1 election: Requested vote from peers 251ddbe523db4a46a9fec795d140e75e (127.1.183.129:40751), 3fecd462e89b46a7920aa054e1675924 (127.1.183.130:42605)
I20260806 02:49:39.021610 29676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36205a0bae1c4d1f971d80305f5699fa" candidate_uuid: "e8f0c9ad2e9b47d781693cea10cf0c83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "251ddbe523db4a46a9fec795d140e75e"
I20260806 02:49:39.021652 29865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36205a0bae1c4d1f971d80305f5699fa" candidate_uuid: "e8f0c9ad2e9b47d781693cea10cf0c83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fecd462e89b46a7920aa054e1675924"
I20260806 02:49:39.021762 29865 raft_consensus.cc:3037] T 36205a0bae1c4d1f971d80305f5699fa P 3fecd462e89b46a7920aa054e1675924 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:39.021754 29676 raft_consensus.cc:3037] T 36205a0bae1c4d1f971d80305f5699fa P 251ddbe523db4a46a9fec795d140e75e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:39.023043 29676 raft_consensus.cc:2454] T 36205a0bae1c4d1f971d80305f5699fa P 251ddbe523db4a46a9fec795d140e75e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8f0c9ad2e9b47d781693cea10cf0c83 in term 1.
I20260806 02:49:39.023043 29865 raft_consensus.cc:2454] T 36205a0bae1c4d1f971d80305f5699fa P 3fecd462e89b46a7920aa054e1675924 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8f0c9ad2e9b47d781693cea10cf0c83 in term 1.
I20260806 02:49:39.023378 29959 leader_election.cc:302] T 36205a0bae1c4d1f971d80305f5699fa P e8f0c9ad2e9b47d781693cea10cf0c83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fecd462e89b46a7920aa054e1675924, e8f0c9ad2e9b47d781693cea10cf0c83; no voters:
I20260806 02:49:39.023546 30173 raft_consensus.cc:2781] T 36205a0bae1c4d1f971d80305f5699fa P e8f0c9ad2e9b47d781693cea10cf0c83 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:49:39.024286 30173 raft_consensus.cc:686] T 36205a0bae1c4d1f971d80305f5699fa P e8f0c9ad2e9b47d781693cea10cf0c83 [term 1 LEADER]: Becoming Leader. State: Replica: e8f0c9ad2e9b47d781693cea10cf0c83, State: Running, Role: LEADER
I20260806 02:49:39.024621 30173 consensus_queue.cc:237] T 36205a0bae1c4d1f971d80305f5699fa P e8f0c9ad2e9b47d781693cea10cf0c83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f0c9ad2e9b47d781693cea10cf0c83" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46593 } } peers { permanent_uuid: "251ddbe523db4a46a9fec795d140e75e" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40751 } } peers { permanent_uuid: "3fecd462e89b46a7920aa054e1675924" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 42605 } }
I20260806 02:49:39.027906 29476 catalog_manager.cc:5495] T 36205a0bae1c4d1f971d80305f5699fa P e8f0c9ad2e9b47d781693cea10cf0c83 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8f0c9ad2e9b47d781693cea10cf0c83 (127.1.183.131). New cstate: current_term: 1 leader_uuid: "e8f0c9ad2e9b47d781693cea10cf0c83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f0c9ad2e9b47d781693cea10cf0c83" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 46593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "251ddbe523db4a46a9fec795d140e75e" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fecd462e89b46a7920aa054e1675924" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 42605 } health_report { overall_health: UNKNOWN } } }
I20260806 02:49:39.042374 30156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:39.038882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34544 (local address 127.1.183.131:46593)
0806 02:49:39.039016 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:39.039019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:39.039033 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:49:39.039163 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:39.039165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:39.039218 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:39.039304 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:49:39.039513 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:39.040008 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:39.040881 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:39.041096 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:39.041174 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:39.041417 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:49:39.041866 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:49:39.041985 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:39.042144 (+ 159us) server_negotiation.cc:299] Negotiation successful
0806 02:49:39.042186 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:49:39.048743 29865 raft_consensus.cc:1261] T 36205a0bae1c4d1f971d80305f5699fa P 3fecd462e89b46a7920aa054e1675924 [term 1 FOLLOWER]: Refusing update from remote peer e8f0c9ad2e9b47d781693cea10cf0c83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:49:39.048738 29676 raft_consensus.cc:1261] T 36205a0bae1c4d1f971d80305f5699fa P 251ddbe523db4a46a9fec795d140e75e [term 1 FOLLOWER]: Refusing update from remote peer e8f0c9ad2e9b47d781693cea10cf0c83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:49:39.049521 30176 consensus_queue.cc:1035] T 36205a0bae1c4d1f971d80305f5699fa P e8f0c9ad2e9b47d781693cea10cf0c83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fecd462e89b46a7920aa054e1675924" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 42605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:49:39.049716 30173 consensus_queue.cc:1035] T 36205a0bae1c4d1f971d80305f5699fa P e8f0c9ad2e9b47d781693cea10cf0c83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "251ddbe523db4a46a9fec795d140e75e" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:49:39.066102 29476 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:49:39.066290 29476 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:49:39.068881 29476 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 79f0ff26aa564005b391dcbed4ea04b1: Sending DeleteTablet for 3 replicas of tablet 36205a0bae1c4d1f971d80305f5699fa
I20260806 02:49:39.069891 29650 tablet_service.cc:1514] Processing DeleteTablet for tablet 36205a0bae1c4d1f971d80305f5699fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:49:39 UTC) from {username='slave'} at 127.0.0.1:34668
I20260806 02:49:39.069895 30032 tablet_service.cc:1514] Processing DeleteTablet for tablet 36205a0bae1c4d1f971d80305f5699fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:49:39 UTC) from {username='slave'} at 127.0.0.1:34536
I20260806 02:49:39.069923 29839 tablet_service.cc:1514] Processing DeleteTablet for tablet 36205a0bae1c4d1f971d80305f5699fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:49:39 UTC) from {username='slave'} at 127.0.0.1:48804
I20260806 02:49:39.070446 30190 tablet_replica.cc:330] T 36205a0bae1c4d1f971d80305f5699fa P 251ddbe523db4a46a9fec795d140e75e: stopping tablet replica
I20260806 02:49:39.070761 30190 raft_consensus.cc:2229] T 36205a0bae1c4d1f971d80305f5699fa P 251ddbe523db4a46a9fec795d140e75e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:49:39.071028 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 29557
I20260806 02:49:39.071137 30190 raft_consensus.cc:2258] T 36205a0bae1c4d1f971d80305f5699fa P 251ddbe523db4a46a9fec795d140e75e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:49:39.076596 30191 tablet_replica.cc:330] T 36205a0bae1c4d1f971d80305f5699fa P 3fecd462e89b46a7920aa054e1675924: stopping tablet replica
I20260806 02:49:39.077061 30191 raft_consensus.cc:2229] T 36205a0bae1c4d1f971d80305f5699fa P 3fecd462e89b46a7920aa054e1675924 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:49:39.077486 30191 raft_consensus.cc:2258] T 36205a0bae1c4d1f971d80305f5699fa P 3fecd462e89b46a7920aa054e1675924 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:49:39.077836 30192 tablet_replica.cc:330] T 36205a0bae1c4d1f971d80305f5699fa P e8f0c9ad2e9b47d781693cea10cf0c83: stopping tablet replica
I20260806 02:49:39.078172 30192 raft_consensus.cc:2229] T 36205a0bae1c4d1f971d80305f5699fa P e8f0c9ad2e9b47d781693cea10cf0c83 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:49:39.078649 30192 raft_consensus.cc:2258] T 36205a0bae1c4d1f971d80305f5699fa P e8f0c9ad2e9b47d781693cea10cf0c83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:49:39.081219 30191 ts_tablet_manager.cc:1905] T 36205a0bae1c4d1f971d80305f5699fa P 3fecd462e89b46a7920aa054e1675924: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:49:39.084012 30191 ts_tablet_manager.cc:1918] T 36205a0bae1c4d1f971d80305f5699fa P 3fecd462e89b46a7920aa054e1675924: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:49:39.084146 30191 log.cc:1196] T 36205a0bae1c4d1f971d80305f5699fa P 3fecd462e89b46a7920aa054e1675924: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-1/wal/wals/36205a0bae1c4d1f971d80305f5699fa
I20260806 02:49:39.084476 30191 ts_tablet_manager.cc:1939] T 36205a0bae1c4d1f971d80305f5699fa P 3fecd462e89b46a7920aa054e1675924: Deleting consensus metadata
I20260806 02:49:39.085739 29456 catalog_manager.cc:4841] TS 3fecd462e89b46a7920aa054e1675924 (127.1.183.130:42605): tablet 36205a0bae1c4d1f971d80305f5699fa (table test-table [id=f152629c387743ae8e19b7c338395337]) successfully deleted
I20260806 02:49:39.086372 30192 ts_tablet_manager.cc:1905] T 36205a0bae1c4d1f971d80305f5699fa P e8f0c9ad2e9b47d781693cea10cf0c83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:49:39.088994 30192 ts_tablet_manager.cc:1918] T 36205a0bae1c4d1f971d80305f5699fa P e8f0c9ad2e9b47d781693cea10cf0c83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:49:39.089114 30192 log.cc:1196] T 36205a0bae1c4d1f971d80305f5699fa P e8f0c9ad2e9b47d781693cea10cf0c83: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984313887056-1758-0/minicluster-data/ts-2/wal/wals/36205a0bae1c4d1f971d80305f5699fa
I20260806 02:49:39.089550 30192 ts_tablet_manager.cc:1939] T 36205a0bae1c4d1f971d80305f5699fa P e8f0c9ad2e9b47d781693cea10cf0c83: Deleting consensus metadata
I20260806 02:49:39.089725 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 29745
W20260806 02:49:39.089704 29458 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.183.129:40751 (error 108)
I20260806 02:49:39.097621 29457 catalog_manager.cc:4841] TS e8f0c9ad2e9b47d781693cea10cf0c83 (127.1.183.131:46593): tablet 36205a0bae1c4d1f971d80305f5699fa (table test-table [id=f152629c387743ae8e19b7c338395337]) successfully deleted
I20260806 02:49:39.098223 30159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:39.097862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:40751 (local address 127.0.0.1:34672)
0806 02:49:39.098069 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:39.098148 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.129:40751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260806 02:49:39.098397 29458 catalog_manager.cc:4584] TS 251ddbe523db4a46a9fec795d140e75e (127.1.183.129:40751): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 36205a0bae1c4d1f971d80305f5699fa: Network error: Client connection negotiation failed: client connection to 127.1.183.129:40751: connect: Connection refused (error 111)
I20260806 02:49:39.104885 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 29938
I20260806 02:49:39.116564 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 29435
2026-08-06T02:49:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1123 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:49:39 dist-test-slave-gmjp krb5kdc[30199](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:49:39 dist-test-slave-gmjp krb5kdc[30199](info): set up 2 sockets
Aug 06 02:49:39 dist-test-slave-gmjp krb5kdc[30199](info): commencing operation
krb5kdc: starting...
W20260806 02:49:40.456782 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.313s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:49:40 dist-test-slave-gmjp krb5kdc[30199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:49:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:49:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
I20260806 02:49:40.591074 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:34691
--webserver_interface=127.1.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:34691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:49:40.705652 30213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:40.705932 30213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:40.705973 30213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:40.709862 30213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:49:40.709913 30213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:40.709932 30213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:40.709949 30213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:49:40.709964 30213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:49:40.715097 30213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:34691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:34691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:40.716117 30213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:40.717273 30213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:40.726109 30221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:40.726217 30213 server_base.cc:1034] running on GCE node
W20260806 02:49:40.726128 30225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:40.726114 30223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:40.727030 30213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:40.728106 30213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:40.729317 30213 hybrid_clock.cc:648] HybridClock initialized: now 1785984580729303 us; error 28 us; skew 500 ppm
Aug 06 02:49:40 dist-test-slave-gmjp krb5kdc[30199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984580, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:40.732123 30213 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:49:40.733035 30213 webserver.cc:460] Webserver started at http://127.1.183.190:37247/ using document root <none> and password file <none>
I20260806 02:49:40.733559 30213 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:40.733618 30213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:40.733783 30213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:40.735795 30213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "753e44c6be1f4ea689c82ebeb2d0db11"
format_stamp: "Formatted at 2026-08-06 02:49:40 on dist-test-slave-gmjp"
server_key: "29abca6961fa63d18f2c084d102f1087"
server_key_iv: "ec9aafb75190742603947b8804401e5f"
server_key_version: "encryptionkey@0"
I20260806 02:49:40.736388 30213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "753e44c6be1f4ea689c82ebeb2d0db11"
format_stamp: "Formatted at 2026-08-06 02:49:40 on dist-test-slave-gmjp"
server_key: "29abca6961fa63d18f2c084d102f1087"
server_key_iv: "ec9aafb75190742603947b8804401e5f"
server_key_version: "encryptionkey@0"
I20260806 02:49:40.740015 30213 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:49:40.742415 30237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:40.743705 30213 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:49:40.743830 30213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "753e44c6be1f4ea689c82ebeb2d0db11"
format_stamp: "Formatted at 2026-08-06 02:49:40 on dist-test-slave-gmjp"
server_key: "29abca6961fa63d18f2c084d102f1087"
server_key_iv: "ec9aafb75190742603947b8804401e5f"
server_key_version: "encryptionkey@0"
I20260806 02:49:40.743940 30213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:40.757930 30213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:40.761334 30213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:40.761557 30213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:40.769160 30213 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:34691
I20260806 02:49:40.769187 30314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:34691 every 8 connection(s)
I20260806 02:49:40.770160 30213 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:49:40.773046 30315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:40.777983 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 30213
I20260806 02:49:40.778151 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:49:40.778486 1758 external_mini_cluster.cc:1428] Setting key 0381e0434bd049fba50622673a053aad
I20260806 02:49:40.779373 30315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 753e44c6be1f4ea689c82ebeb2d0db11: Bootstrap starting.
I20260806 02:49:40.782043 30315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 753e44c6be1f4ea689c82ebeb2d0db11: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:40.783105 30315 log.cc:824] T 00000000000000000000000000000000 P 753e44c6be1f4ea689c82ebeb2d0db11: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:40.785770 30315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 753e44c6be1f4ea689c82ebeb2d0db11: No bootstrap required, opened a new log
Aug 06 02:49:40 dist-test-slave-gmjp krb5kdc[30199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:49:40.790634 30315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 753e44c6be1f4ea689c82ebeb2d0db11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "753e44c6be1f4ea689c82ebeb2d0db11" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34691 } }
I20260806 02:49:40.790930 30315 raft_consensus.cc:374] T 00000000000000000000000000000000 P 753e44c6be1f4ea689c82ebeb2d0db11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:40.791014 30315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 753e44c6be1f4ea689c82ebeb2d0db11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 753e44c6be1f4ea689c82ebeb2d0db11, State: Initialized, Role: FOLLOWER
I20260806 02:49:40.791538 30315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 753e44c6be1f4ea689c82ebeb2d0db11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "753e44c6be1f4ea689c82ebeb2d0db11" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34691 } }
I20260806 02:49:40.791702 30315 raft_consensus.cc:388] T 00000000000000000000000000000000 P 753e44c6be1f4ea689c82ebeb2d0db11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:49:40.791797 30315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 753e44c6be1f4ea689c82ebeb2d0db11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:49:40.791927 30315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 753e44c6be1f4ea689c82ebeb2d0db11 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:40.793267 30315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 753e44c6be1f4ea689c82ebeb2d0db11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "753e44c6be1f4ea689c82ebeb2d0db11" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34691 } }
I20260806 02:49:40.793696 30315 leader_election.cc:302] T 00000000000000000000000000000000 P 753e44c6be1f4ea689c82ebeb2d0db11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 753e44c6be1f4ea689c82ebeb2d0db11; no voters:
I20260806 02:49:40.794075 30315 leader_election.cc:288] T 00000000000000000000000000000000 P 753e44c6be1f4ea689c82ebeb2d0db11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:49:40.794189 30322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 753e44c6be1f4ea689c82ebeb2d0db11 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:49:40.794483 30322 raft_consensus.cc:686] T 00000000000000000000000000000000 P 753e44c6be1f4ea689c82ebeb2d0db11 [term 1 LEADER]: Becoming Leader. State: Replica: 753e44c6be1f4ea689c82ebeb2d0db11, State: Running, Role: LEADER
I20260806 02:49:40.794878 30322 consensus_queue.cc:237] T 00000000000000000000000000000000 P 753e44c6be1f4ea689c82ebeb2d0db11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "753e44c6be1f4ea689c82ebeb2d0db11" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34691 } }
I20260806 02:49:40.794981 30320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:40.780137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57566 (local address 127.1.183.190:34691)
0806 02:49:40.780635 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:40.780652 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:40.780688 (+ 36us) server_negotiation.cc:407] Connection header received
0806 02:49:40.781399 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:40.781420 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:40.781764 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:40.782076 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:40.783086 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:40.784543 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:40.785736 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:40.786280 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:40.786399 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:40.789423 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:40.789428 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:40.789443 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:40.789472 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:40.791333 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:40.791854 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:40.791861 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:40.791868 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:40.792033 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:40.792374 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:40.792378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:40.792380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:40.792960 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:40.793183 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:40.793522 (+ 339us) server_negotiation.cc:299] Negotiation successful
0806 02:49:40.793864 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":148,"threads_started":1}
I20260806 02:49:40.795471 30315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 753e44c6be1f4ea689c82ebeb2d0db11 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:49:40.798543 30324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 753e44c6be1f4ea689c82ebeb2d0db11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "753e44c6be1f4ea689c82ebeb2d0db11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "753e44c6be1f4ea689c82ebeb2d0db11" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34691 } } }
I20260806 02:49:40.798640 30324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 753e44c6be1f4ea689c82ebeb2d0db11 [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:40.798285 30323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 753e44c6be1f4ea689c82ebeb2d0db11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 753e44c6be1f4ea689c82ebeb2d0db11. Latest consensus state: current_term: 1 leader_uuid: "753e44c6be1f4ea689c82ebeb2d0db11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "753e44c6be1f4ea689c82ebeb2d0db11" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34691 } } }
I20260806 02:49:40.798841 30323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 753e44c6be1f4ea689c82ebeb2d0db11 [sys.catalog]: This master's current role is: LEADER
W20260806 02:49:40.800693 30342 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 753e44c6be1f4ea689c82ebeb2d0db11: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:49:40.800784 30342 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:49:40.800863 30345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:49:40.801770 30345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:49:40.807595 30345 catalog_manager.cc:1347] Generated new cluster ID: 24eece9ce7eb48998b498491de77aa8a
I20260806 02:49:40.807704 30345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:49:40.818720 30345 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:49:40.819672 30345 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:49:40.825364 30345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 753e44c6be1f4ea689c82ebeb2d0db11: Generated new TSK 0
I20260806 02:49:40.826079 30345 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.183.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
Entry for principal kudu/127.1.183.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab.
I20260806 02:49:40.865309 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.129:0
--local_ip_for_outbound_sockets=127.1.183.129
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:34691
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:49:40.980609 30353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:40.980916 30353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:40.980991 30353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:40.985039 30353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:40.985134 30353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:40.985239 30353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.129
I20260806 02:49:40.990577 30353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.129.keytab
--principal=kudu/127.1.183.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:34691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.129
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:40.991767 30353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:40.992864 30353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:41.003067 30361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:41.003178 30360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:41.003280 30353 server_base.cc:1034] running on GCE node
W20260806 02:49:41.003069 30364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:41.003847 30353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:41.004585 30353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:41.005748 30353 hybrid_clock.cc:648] HybridClock initialized: now 1785984581005733 us; error 27 us; skew 500 ppm
Aug 06 02:49:41 dist-test-slave-gmjp krb5kdc[30199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984581, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:41.008549 30353 init.cc:377] Logged in from keytab as kudu/127.1.183.129@KRBTEST.COM (short username kudu)
I20260806 02:49:41.009413 30353 webserver.cc:460] Webserver started at http://127.1.183.129:33687/ using document root <none> and password file <none>
I20260806 02:49:41.010007 30353 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:41.010106 30353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:41.010339 30353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:41.012534 30353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-0/data/instance:
uuid: "81cee2d04b064bf68ffc64734307cbe6"
format_stamp: "Formatted at 2026-08-06 02:49:41 on dist-test-slave-gmjp"
server_key: "0575ce513fc570cb2977f089dc9af4e7"
server_key_iv: "47e46e6b90d4f0e37767692235a08739"
server_key_version: "encryptionkey@0"
I20260806 02:49:41.013121 30353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance:
uuid: "81cee2d04b064bf68ffc64734307cbe6"
format_stamp: "Formatted at 2026-08-06 02:49:41 on dist-test-slave-gmjp"
server_key: "0575ce513fc570cb2977f089dc9af4e7"
server_key_iv: "47e46e6b90d4f0e37767692235a08739"
server_key_version: "encryptionkey@0"
I20260806 02:49:41.016886 30353 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:49:41.019363 30372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:41.020692 30353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 02:49:41.020833 30353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-0/wal
uuid: "81cee2d04b064bf68ffc64734307cbe6"
format_stamp: "Formatted at 2026-08-06 02:49:41 on dist-test-slave-gmjp"
server_key: "0575ce513fc570cb2977f089dc9af4e7"
server_key_iv: "47e46e6b90d4f0e37767692235a08739"
server_key_version: "encryptionkey@0"
I20260806 02:49:41.020951 30353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:41.039583 30353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:41.042589 30353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:41.042814 30353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:41.043462 30353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:49:41.044399 30353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:49:41.044481 30353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:41.044538 30353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:49:41.044569 30353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:41.054356 30353 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.129:41609
I20260806 02:49:41.054386 30540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.129:41609 every 8 connection(s)
I20260806 02:49:41.055353 30353 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
I20260806 02:49:41.062983 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 30353
I20260806 02:49:41.063133 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance
I20260806 02:49:41.063516 1758 external_mini_cluster.cc:1428] Setting key 2f5fe47b15ef5ae1035ddaa3f6b0decd
Aug 06 02:49:41 dist-test-slave-gmjp krb5kdc[30199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984581, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.129@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:49:41.070134 30320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:41.057298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:49473 (local address 127.1.183.190:34691)
0806 02:49:41.057476 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:41.057481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:41.058332 (+ 851us) server_negotiation.cc:407] Connection header received
0806 02:49:41.059612 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:41.059615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:41.059663 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:41.059737 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:41.060982 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.061504 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.062594 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.062813 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.062905 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:41.065916 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:41.065923 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:41.065929 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:41.065962 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:41.067624 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:41.068289 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:41.068292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:41.068293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:41.068369 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:41.068788 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:41.068790 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:41.068791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:41.069049 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:41.069156 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:41.069682 (+ 526us) server_negotiation.cc:299] Negotiation successful
0806 02:49:41.069845 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:49:41.070824 30543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:41.057726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34691 (local address 127.1.183.129:49473)
0806 02:49:41.058170 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:41.058214 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:41.059411 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:41.059901 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:41.059908 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:41.060345 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:49:41.060842 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:41.060856 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.061612 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.061614 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:41.062484 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:41.062493 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.062988 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.062992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:41.063212 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:41.063886 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:49:41.063910 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:49:41.065718 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:49:41.067750 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:41.067757 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:41.067775 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:41.068188 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:41.068466 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:41.068471 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:41.068472 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:41.068678 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:41.069160 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:49:41.069165 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:49:41.069421 (+ 256us) client_negotiation.cc:769] Sending connection context
0806 02:49:41.069579 (+ 158us) client_negotiation.cc:240] Negotiation successful
0806 02:49:41.069850 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":118,"threads_started":1}
I20260806 02:49:41.072185 30541 heartbeater.cc:347] Connected to a master server at 127.1.183.190:34691
I20260806 02:49:41.072475 30541 heartbeater.cc:464] Registering TS with master...
I20260806 02:49:41.073127 30541 heartbeater.cc:511] Master 127.1.183.190:34691 requested a full tablet report, sending...
I20260806 02:49:41.074741 30259 ts_manager.cc:194] Registered new tserver with Master: 81cee2d04b064bf68ffc64734307cbe6 (127.1.183.129:41609)
WARNING: no policy specified for kudu/127.1.183.130@KRBTEST.COM; defaulting to no policy
I20260806 02:49:41.075960 30259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.129@KRBTEST.COM'} at 127.1.183.129:49473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
Entry for principal kudu/127.1.183.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab.
I20260806 02:49:41.094658 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.130:0
--local_ip_for_outbound_sockets=127.1.183.130
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:34691
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:49:41.217128 30548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:41.217396 30548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:41.217475 30548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:41.221710 30548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:41.221778 30548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:41.221839 30548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.130
I20260806 02:49:41.227474 30548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.130.keytab
--principal=kudu/127.1.183.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:34691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.130
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:41.228649 30548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:41.229736 30548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:41.239614 30562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:41.239614 30560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:41.239640 30564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:41.239971 30548 server_base.cc:1034] running on GCE node
I20260806 02:49:41.240399 30548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:41.240974 30548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:41.242136 30548 hybrid_clock.cc:648] HybridClock initialized: now 1785984581242117 us; error 26 us; skew 500 ppm
Aug 06 02:49:41 dist-test-slave-gmjp krb5kdc[30199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984581, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:41.244979 30548 init.cc:377] Logged in from keytab as kudu/127.1.183.130@KRBTEST.COM (short username kudu)
I20260806 02:49:41.245779 30548 webserver.cc:460] Webserver started at http://127.1.183.130:38575/ using document root <none> and password file <none>
I20260806 02:49:41.246294 30548 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:41.246353 30548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:41.246523 30548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:41.248678 30548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-1/data/instance:
uuid: "6982b5bc2eec4c758d9321d73012474f"
format_stamp: "Formatted at 2026-08-06 02:49:41 on dist-test-slave-gmjp"
server_key: "12aee8250b741a0a3cc2ea48c275c28f"
server_key_iv: "044749cdafa9ea00b952ea5ba5ec0888"
server_key_version: "encryptionkey@0"
I20260806 02:49:41.249190 30548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance:
uuid: "6982b5bc2eec4c758d9321d73012474f"
format_stamp: "Formatted at 2026-08-06 02:49:41 on dist-test-slave-gmjp"
server_key: "12aee8250b741a0a3cc2ea48c275c28f"
server_key_iv: "044749cdafa9ea00b952ea5ba5ec0888"
server_key_version: "encryptionkey@0"
I20260806 02:49:41.252801 30548 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:49:41.255124 30574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:41.256372 30548 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:49:41.256511 30548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-1/wal
uuid: "6982b5bc2eec4c758d9321d73012474f"
format_stamp: "Formatted at 2026-08-06 02:49:41 on dist-test-slave-gmjp"
server_key: "12aee8250b741a0a3cc2ea48c275c28f"
server_key_iv: "044749cdafa9ea00b952ea5ba5ec0888"
server_key_version: "encryptionkey@0"
I20260806 02:49:41.256620 30548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:41.273082 30548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:41.276420 30548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:41.276628 30548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:41.277233 30548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:49:41.278084 30548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:49:41.278157 30548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:41.278230 30548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:49:41.278281 30548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:41.287566 30548 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.130:34193
I20260806 02:49:41.287595 30728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.130:34193 every 8 connection(s)
I20260806 02:49:41.288496 30548 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
I20260806 02:49:41.292618 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 30548
I20260806 02:49:41.292784 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance
I20260806 02:49:41.293133 1758 external_mini_cluster.cc:1428] Setting key 3884c20f215e302016e8c062e85fe8a5
Aug 06 02:49:41 dist-test-slave-gmjp krb5kdc[30199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984581, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.130@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:49:41.305179 30320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:41.290502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:53493 (local address 127.1.183.190:34691)
0806 02:49:41.290653 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:41.290657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:41.291517 (+ 860us) server_negotiation.cc:407] Connection header received
0806 02:49:41.293301 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:41.293305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:41.293375 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:41.293462 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:41.295018 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.295629 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.297549 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.297942 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.298070 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:41.301126 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:41.301133 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:41.301140 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:41.301177 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:41.302915 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:41.303452 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:41.303456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:41.303458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:41.303559 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:41.303905 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:41.303910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:41.303912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:41.304221 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:41.304330 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:41.304863 (+ 533us) server_negotiation.cc:299] Negotiation successful
0806 02:49:41.305004 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
WARNING: no policy specified for kudu/127.1.183.131@KRBTEST.COM; defaulting to no policy
I20260806 02:49:41.306003 30733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:41.290887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34691 (local address 127.1.183.130:53493)
0806 02:49:41.291346 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:41.291401 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:41.292999 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:41.293695 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:41.293703 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:41.294251 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:49:41.294854 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:41.294877 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.295766 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.295773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:41.297395 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:41.297407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.298128 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.298135 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:41.298322 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:41.299060 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:49:41.299088 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:49:41.300932 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:49:41.303035 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:41.303041 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:41.303056 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:41.303329 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:41.303657 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:41.303660 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:41.303661 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:41.303796 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:41.304323 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:49:41.304329 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:49:41.304582 (+ 253us) client_negotiation.cc:769] Sending connection context
0806 02:49:41.304770 (+ 188us) client_negotiation.cc:240] Negotiation successful
0806 02:49:41.305056 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":119,"threads_started":1}
I20260806 02:49:41.307264 30730 heartbeater.cc:347] Connected to a master server at 127.1.183.190:34691
I20260806 02:49:41.307574 30730 heartbeater.cc:464] Registering TS with master...
I20260806 02:49:41.308115 30730 heartbeater.cc:511] Master 127.1.183.190:34691 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.131@KRBTEST.COM" created.
I20260806 02:49:41.309397 30259 ts_manager.cc:194] Registered new tserver with Master: 6982b5bc2eec4c758d9321d73012474f (127.1.183.130:34193)
I20260806 02:49:41.310011 30259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.130@KRBTEST.COM'} at 127.1.183.130:53493
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
Entry for principal kudu/127.1.183.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab.
I20260806 02:49:41.324857 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.131:0
--local_ip_for_outbound_sockets=127.1.183.131
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:34691
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:44843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:49:41.442327 30739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:41.442595 30739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:41.442637 30739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:41.447068 30739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:41.447129 30739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:41.447190 30739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.131
I20260806 02:49:41.452642 30739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:44843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.131.keytab
--principal=kudu/127.1.183.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:34691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.131
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:41.453649 30739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:41.454641 30739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:41.464571 30751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:41.464627 30758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:41.464598 30748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:41.464800 30739 server_base.cc:1034] running on GCE node
I20260806 02:49:41.465176 30739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:41.465831 30739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:41.467020 30739 hybrid_clock.cc:648] HybridClock initialized: now 1785984581466996 us; error 53 us; skew 500 ppm
Aug 06 02:49:41 dist-test-slave-gmjp krb5kdc[30199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984581, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:41.469712 30739 init.cc:377] Logged in from keytab as kudu/127.1.183.131@KRBTEST.COM (short username kudu)
I20260806 02:49:41.470551 30739 webserver.cc:460] Webserver started at http://127.1.183.131:42521/ using document root <none> and password file <none>
I20260806 02:49:41.471107 30739 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:41.471197 30739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:41.471436 30739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:41.473491 30739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-2/data/instance:
uuid: "ed6ef968f3ee4a8cb2aa60e0a97dcfd7"
format_stamp: "Formatted at 2026-08-06 02:49:41 on dist-test-slave-gmjp"
server_key: "0c17fb477884e774e095b13295926b28"
server_key_iv: "a8cd0071d9659964fa204cde808bc059"
server_key_version: "encryptionkey@0"
I20260806 02:49:41.474052 30739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance:
uuid: "ed6ef968f3ee4a8cb2aa60e0a97dcfd7"
format_stamp: "Formatted at 2026-08-06 02:49:41 on dist-test-slave-gmjp"
server_key: "0c17fb477884e774e095b13295926b28"
server_key_iv: "a8cd0071d9659964fa204cde808bc059"
server_key_version: "encryptionkey@0"
I20260806 02:49:41.477692 30739 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 02:49:41.480042 30774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:41.481423 30739 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:49:41.481550 30739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-2/wal
uuid: "ed6ef968f3ee4a8cb2aa60e0a97dcfd7"
format_stamp: "Formatted at 2026-08-06 02:49:41 on dist-test-slave-gmjp"
server_key: "0c17fb477884e774e095b13295926b28"
server_key_iv: "a8cd0071d9659964fa204cde808bc059"
server_key_version: "encryptionkey@0"
I20260806 02:49:41.481669 30739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:41.506238 30739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:41.509145 30739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:41.509368 30739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:41.509981 30739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:49:41.510857 30739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:49:41.510941 30739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:41.511037 30739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:49:41.511088 30739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:41.520423 30739 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.131:37341
I20260806 02:49:41.520444 30931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.131:37341 every 8 connection(s)
I20260806 02:49:41.521343 30739 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
I20260806 02:49:41.522142 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 30739
I20260806 02:49:41.522289 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance
I20260806 02:49:41.522651 1758 external_mini_cluster.cc:1428] Setting key 263dd16d52aecd5ecabf9b18bfb84102
Aug 06 02:49:41 dist-test-slave-gmjp krb5kdc[30199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984581, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.131@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:49:41.535303 30320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:41.523563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:55091 (local address 127.1.183.190:34691)
0806 02:49:41.523706 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:41.523716 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:41.524818 (+ 1102us) server_negotiation.cc:407] Connection header received
0806 02:49:41.526333 (+ 1515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:41.526336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:41.526384 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:41.526463 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:41.527613 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.528198 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.529300 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.529518 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.529599 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:41.532003 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:41.532006 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:41.532009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:41.532035 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:41.533549 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:41.534004 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:41.534007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:41.534008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:41.534079 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:41.534400 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:41.534402 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:41.534403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:41.534566 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:41.534642 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:41.535085 (+ 443us) server_negotiation.cc:299] Negotiation successful
0806 02:49:41.535178 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:49:41.536419 30934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:41.524009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34691 (local address 127.1.183.131:55091)
0806 02:49:41.524649 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:41.524712 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:41.526128 (+ 1416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:41.526597 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:41.526604 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:41.526968 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:49:41.527472 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:41.527486 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.528308 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.528311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:41.529192 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:41.529202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.529654 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.529657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:41.529805 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:41.530340 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:49:41.530356 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:49:41.531836 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:49:41.533654 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:41.533658 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:41.533670 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:41.533919 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:41.534159 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:49:41.534161 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:49:41.534162 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:49:41.534304 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:49:41.534655 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:49:41.534660 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:49:41.534884 (+ 224us) client_negotiation.cc:769] Sending connection context
0806 02:49:41.535038 (+ 154us) client_negotiation.cc:240] Negotiation successful
0806 02:49:41.535381 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":140,"threads_started":1}
I20260806 02:49:41.537513 30932 heartbeater.cc:347] Connected to a master server at 127.1.183.190:34691
I20260806 02:49:41.537753 30932 heartbeater.cc:464] Registering TS with master...
I20260806 02:49:41.538257 30932 heartbeater.cc:511] Master 127.1.183.190:34691 requested a full tablet report, sending...
I20260806 02:49:41.539295 30259 ts_manager.cc:194] Registered new tserver with Master: ed6ef968f3ee4a8cb2aa60e0a97dcfd7 (127.1.183.131:37341)
I20260806 02:49:41.539860 30259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.183.131@KRBTEST.COM'} at 127.1.183.131:55091
I20260806 02:49:41.548874 1758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:49:41.558167 30320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:41.550954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57578 (local address 127.1.183.190:34691)
0806 02:49:41.551202 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:41.551207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:41.551351 (+ 144us) server_negotiation.cc:407] Connection header received
0806 02:49:41.551465 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:41.551467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:41.551510 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:41.551578 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:41.552324 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.552831 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.553631 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.553831 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.553905 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:41.555173 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:41.555176 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:41.555178 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:41.555201 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:41.556691 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:41.557056 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:41.557058 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:41.557060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:41.557134 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:41.557361 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:41.557363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:41.557364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:41.557582 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:41.557688 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:41.557857 (+ 169us) server_negotiation.cc:299] Negotiation successful
0806 02:49:41.557951 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260806 02:49:41.561192 30259 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:49:41.564256 30259 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:49:41.579246 30946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:41.572908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:34193 (local address 127.0.0.1:42726)
0806 02:49:41.573745 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:41.573797 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:41.573937 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:41.574841 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:41.574847 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:41.574886 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:41.575061 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:41.575069 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.576100 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.576105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:41.577537 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:41.577548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.578690 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.578694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:41.578787 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:41.578847 (+ 60us) client_negotiation.cc:769] Sending connection context
0806 02:49:41.578930 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 02:49:41.579049 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":38,"spinlock_wait_cycles":7296,"thread_start_us":69,"threads_started":1}
I20260806 02:49:41.579276 30947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:41.573054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42726 (local address 127.1.183.130:34193)
0806 02:49:41.574355 (+ 1301us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:41.574368 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:41.574402 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:49:41.574491 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:41.574495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:41.574653 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:41.574984 (+ 331us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:41.575158 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.575968 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.577677 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.578546 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.578847 (+ 301us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:41.578994 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:41.579049 (+ 55us) server_negotiation.cc:299] Negotiation successful
0806 02:49:41.579141 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1171,"thread_start_us":121,"threads_started":1}
I20260806 02:49:41.579432 30944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:41.572976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:41609 (local address 127.0.0.1:34692)
0806 02:49:41.574246 (+ 1270us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:41.574260 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:41.574361 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:41.574841 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:41.574847 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:41.574885 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:41.575238 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:41.575243 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.576637 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.576641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:41.577833 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:41.577841 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.578636 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.578640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:41.578759 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:41.579165 (+ 406us) client_negotiation.cc:769] Sending connection context
0806 02:49:41.579233 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 02:49:41.579298 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1199,"mutex_wait_us":120,"spinlock_wait_cycles":5760,"thread_start_us":103,"threads_started":1}
I20260806 02:49:41.579707 30948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:41.573258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:37341 (local address 127.0.0.1:47098)
0806 02:49:41.574282 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:41.574294 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:41.574423 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:41.575040 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:41.575042 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:41.575056 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:41.575134 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:41.575138 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.576637 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.576641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:41.577562 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:41.577572 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.578861 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.578865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:41.578945 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:41.578966 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 02:49:41.579049 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 02:49:41.579120 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":972,"mutex_wait_us":48,"thread_start_us":91,"threads_started":1}
I20260806 02:49:41.579708 30951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:41.573702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47098 (local address 127.1.183.131:37341)
0806 02:49:41.574155 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:41.574164 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:41.574524 (+ 360us) server_negotiation.cc:407] Connection header received
0806 02:49:41.574600 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:41.574605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:41.574759 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:41.574894 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:41.575245 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.576472 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.577725 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.578546 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.578664 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:41.578805 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:41.579105 (+ 300us) server_negotiation.cc:299] Negotiation successful
0806 02:49:41.579212 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":176,"threads_started":1}
I20260806 02:49:41.579933 30949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:41.573701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34692 (local address 127.1.183.129:41609)
0806 02:49:41.574088 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:41.574096 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:41.574384 (+ 288us) server_negotiation.cc:407] Connection header received
0806 02:49:41.574475 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:41.574481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:41.574676 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:41.574824 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:41.575357 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.576495 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.577953 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.578505 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.578633 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:41.578788 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:41.579673 (+ 885us) server_negotiation.cc:299] Negotiation successful
0806 02:49:41.579781 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":90,"threads_started":1}
I20260806 02:49:41.582511 30641 tablet_service.cc:1467] Processing CreateTablet for tablet b06a11565be84c10a9a4962223f45e48 (DEFAULT_TABLE table=test-table [id=2cc2a5567b5743e6a8bb7f6692b01448]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:49:41.582758 30447 tablet_service.cc:1467] Processing CreateTablet for tablet b06a11565be84c10a9a4962223f45e48 (DEFAULT_TABLE table=test-table [id=2cc2a5567b5743e6a8bb7f6692b01448]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:49:41.582758 30834 tablet_service.cc:1467] Processing CreateTablet for tablet b06a11565be84c10a9a4962223f45e48 (DEFAULT_TABLE table=test-table [id=2cc2a5567b5743e6a8bb7f6692b01448]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:49:41.583557 30641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b06a11565be84c10a9a4962223f45e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:41.583786 30834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b06a11565be84c10a9a4962223f45e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:41.583781 30447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b06a11565be84c10a9a4962223f45e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:41.590354 30953 tablet_bootstrap.cc:492] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f: Bootstrap starting.
I20260806 02:49:41.591812 30955 tablet_bootstrap.cc:492] T b06a11565be84c10a9a4962223f45e48 P ed6ef968f3ee4a8cb2aa60e0a97dcfd7: Bootstrap starting.
I20260806 02:49:41.591812 30954 tablet_bootstrap.cc:492] T b06a11565be84c10a9a4962223f45e48 P 81cee2d04b064bf68ffc64734307cbe6: Bootstrap starting.
I20260806 02:49:41.593091 30953 tablet_bootstrap.cc:654] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:41.593870 30953 log.cc:824] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:41.594097 30954 tablet_bootstrap.cc:654] T b06a11565be84c10a9a4962223f45e48 P 81cee2d04b064bf68ffc64734307cbe6: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:41.594097 30955 tablet_bootstrap.cc:654] T b06a11565be84c10a9a4962223f45e48 P ed6ef968f3ee4a8cb2aa60e0a97dcfd7: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:41.595045 30954 log.cc:824] T b06a11565be84c10a9a4962223f45e48 P 81cee2d04b064bf68ffc64734307cbe6: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:41.595417 30955 log.cc:824] T b06a11565be84c10a9a4962223f45e48 P ed6ef968f3ee4a8cb2aa60e0a97dcfd7: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:41.596383 30953 tablet_bootstrap.cc:492] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f: No bootstrap required, opened a new log
I20260806 02:49:41.596623 30953 ts_tablet_manager.cc:1397] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:49:41.597472 30955 tablet_bootstrap.cc:492] T b06a11565be84c10a9a4962223f45e48 P ed6ef968f3ee4a8cb2aa60e0a97dcfd7: No bootstrap required, opened a new log
I20260806 02:49:41.597472 30954 tablet_bootstrap.cc:492] T b06a11565be84c10a9a4962223f45e48 P 81cee2d04b064bf68ffc64734307cbe6: No bootstrap required, opened a new log
I20260806 02:49:41.597673 30954 ts_tablet_manager.cc:1397] T b06a11565be84c10a9a4962223f45e48 P 81cee2d04b064bf68ffc64734307cbe6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 02:49:41.597673 30955 ts_tablet_manager.cc:1397] T b06a11565be84c10a9a4962223f45e48 P ed6ef968f3ee4a8cb2aa60e0a97dcfd7: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:49:41.599853 30953 raft_consensus.cc:348] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6982b5bc2eec4c758d9321d73012474f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34193 } } peers { permanent_uuid: "81cee2d04b064bf68ffc64734307cbe6" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 41609 } } peers { permanent_uuid: "ed6ef968f3ee4a8cb2aa60e0a97dcfd7" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37341 } }
I20260806 02:49:41.600160 30953 raft_consensus.cc:374] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:41.600255 30953 raft_consensus.cc:729] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6982b5bc2eec4c758d9321d73012474f, State: Initialized, Role: FOLLOWER
I20260806 02:49:41.600751 30953 consensus_queue.cc:260] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6982b5bc2eec4c758d9321d73012474f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34193 } } peers { permanent_uuid: "81cee2d04b064bf68ffc64734307cbe6" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 41609 } } peers { permanent_uuid: "ed6ef968f3ee4a8cb2aa60e0a97dcfd7" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37341 } }
I20260806 02:49:41.601464 30730 heartbeater.cc:503] Master 127.1.183.190:34691 was elected leader, sending a full tablet report...
I20260806 02:49:41.602157 30954 raft_consensus.cc:348] T b06a11565be84c10a9a4962223f45e48 P 81cee2d04b064bf68ffc64734307cbe6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6982b5bc2eec4c758d9321d73012474f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34193 } } peers { permanent_uuid: "81cee2d04b064bf68ffc64734307cbe6" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 41609 } } peers { permanent_uuid: "ed6ef968f3ee4a8cb2aa60e0a97dcfd7" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37341 } }
I20260806 02:49:41.602084 30955 raft_consensus.cc:348] T b06a11565be84c10a9a4962223f45e48 P ed6ef968f3ee4a8cb2aa60e0a97dcfd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6982b5bc2eec4c758d9321d73012474f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34193 } } peers { permanent_uuid: "81cee2d04b064bf68ffc64734307cbe6" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 41609 } } peers { permanent_uuid: "ed6ef968f3ee4a8cb2aa60e0a97dcfd7" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37341 } }
I20260806 02:49:41.602388 30955 raft_consensus.cc:374] T b06a11565be84c10a9a4962223f45e48 P ed6ef968f3ee4a8cb2aa60e0a97dcfd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:41.602432 30954 raft_consensus.cc:374] T b06a11565be84c10a9a4962223f45e48 P 81cee2d04b064bf68ffc64734307cbe6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:41.602470 30955 raft_consensus.cc:729] T b06a11565be84c10a9a4962223f45e48 P ed6ef968f3ee4a8cb2aa60e0a97dcfd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed6ef968f3ee4a8cb2aa60e0a97dcfd7, State: Initialized, Role: FOLLOWER
I20260806 02:49:41.602519 30954 raft_consensus.cc:729] T b06a11565be84c10a9a4962223f45e48 P 81cee2d04b064bf68ffc64734307cbe6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81cee2d04b064bf68ffc64734307cbe6, State: Initialized, Role: FOLLOWER
I20260806 02:49:41.602955 30954 consensus_queue.cc:260] T b06a11565be84c10a9a4962223f45e48 P 81cee2d04b064bf68ffc64734307cbe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6982b5bc2eec4c758d9321d73012474f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34193 } } peers { permanent_uuid: "81cee2d04b064bf68ffc64734307cbe6" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 41609 } } peers { permanent_uuid: "ed6ef968f3ee4a8cb2aa60e0a97dcfd7" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37341 } }
I20260806 02:49:41.602955 30955 consensus_queue.cc:260] T b06a11565be84c10a9a4962223f45e48 P ed6ef968f3ee4a8cb2aa60e0a97dcfd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6982b5bc2eec4c758d9321d73012474f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34193 } } peers { permanent_uuid: "81cee2d04b064bf68ffc64734307cbe6" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 41609 } } peers { permanent_uuid: "ed6ef968f3ee4a8cb2aa60e0a97dcfd7" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37341 } }
I20260806 02:49:41.603235 30953 ts_tablet_manager.cc:1428] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 02:49:41.603948 30955 ts_tablet_manager.cc:1428] T b06a11565be84c10a9a4962223f45e48 P ed6ef968f3ee4a8cb2aa60e0a97dcfd7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 02:49:41.603948 30954 ts_tablet_manager.cc:1428] T b06a11565be84c10a9a4962223f45e48 P 81cee2d04b064bf68ffc64734307cbe6: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260806 02:49:41.604337 30932 heartbeater.cc:503] Master 127.1.183.190:34691 was elected leader, sending a full tablet report...
I20260806 02:49:41.604337 30541 heartbeater.cc:503] Master 127.1.183.190:34691 was elected leader, sending a full tablet report...
W20260806 02:49:41.772388 30933 tablet.cc:2369] T b06a11565be84c10a9a4962223f45e48 P ed6ef968f3ee4a8cb2aa60e0a97dcfd7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:49:41.789778 30731 tablet.cc:2369] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:49:41.806756 30542 tablet.cc:2369] T b06a11565be84c10a9a4962223f45e48 P 81cee2d04b064bf68ffc64734307cbe6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:49:41.908896 30961 raft_consensus.cc:482] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:49:41.909174 30961 raft_consensus.cc:504] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6982b5bc2eec4c758d9321d73012474f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34193 } } peers { permanent_uuid: "81cee2d04b064bf68ffc64734307cbe6" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 41609 } } peers { permanent_uuid: "ed6ef968f3ee4a8cb2aa60e0a97dcfd7" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37341 } }
I20260806 02:49:41.910466 30961 leader_election.cc:288] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81cee2d04b064bf68ffc64734307cbe6 (127.1.183.129:41609), ed6ef968f3ee4a8cb2aa60e0a97dcfd7 (127.1.183.131:37341)
I20260806 02:49:41.914713 30951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:41.910894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:42277 (local address 127.1.183.131:37341)
0806 02:49:41.911032 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:41.911037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:41.911230 (+ 193us) server_negotiation.cc:407] Connection header received
0806 02:49:41.911436 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:41.911439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:41.911480 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:41.911570 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:41.911882 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.912420 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.913656 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.914081 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.914220 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:41.914277 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:41.914476 (+ 199us) server_negotiation.cc:299] Negotiation successful
0806 02:49:41.914562 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:49:41.914729 30949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:41.910880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:54311 (local address 127.1.183.129:41609)
0806 02:49:41.911018 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:41.911024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:41.911232 (+ 208us) server_negotiation.cc:407] Connection header received
0806 02:49:41.911434 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:41.911437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:41.911479 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:41.911567 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:41.911916 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.912469 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.913654 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.914103 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.914239 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:41.914291 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:41.914487 (+ 196us) server_negotiation.cc:299] Negotiation successful
0806 02:49:41.914584 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:49:41.914760 30968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:41.910753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:41609 (local address 127.1.183.130:54311)
0806 02:49:41.911119 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:41.911131 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:41.911311 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:41.911583 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:41.911586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:41.911615 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:41.911796 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:41.911802 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.912585 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.912589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:41.913516 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:41.913526 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.914239 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.914243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:41.914349 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:41.914371 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:49:41.914469 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 02:49:41.914601 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":75,"spinlock_wait_cycles":18176,"thread_start_us":125,"threads_started":1}
I20260806 02:49:41.914773 30966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:41.910769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:37341 (local address 127.1.183.130:42277)
0806 02:49:41.911129 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:41.911141 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:41.911307 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:41.911583 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:41.911586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:41.911617 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:41.911759 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:41.911764 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.912539 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.912544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:41.913519 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:41.913529 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.914217 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.914221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:41.914323 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:41.914355 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 02:49:41.914456 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 02:49:41.914584 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":96,"spinlock_wait_cycles":37888,"thread_start_us":85,"threads_started":1}
I20260806 02:49:41.915494 30479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b06a11565be84c10a9a4962223f45e48" candidate_uuid: "6982b5bc2eec4c758d9321d73012474f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81cee2d04b064bf68ffc64734307cbe6" is_pre_election: true
I20260806 02:49:41.915648 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b06a11565be84c10a9a4962223f45e48" candidate_uuid: "6982b5bc2eec4c758d9321d73012474f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed6ef968f3ee4a8cb2aa60e0a97dcfd7" is_pre_election: true
I20260806 02:49:41.915865 30479 raft_consensus.cc:2454] T b06a11565be84c10a9a4962223f45e48 P 81cee2d04b064bf68ffc64734307cbe6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6982b5bc2eec4c758d9321d73012474f in term 0.
I20260806 02:49:41.916019 30863 raft_consensus.cc:2454] T b06a11565be84c10a9a4962223f45e48 P ed6ef968f3ee4a8cb2aa60e0a97dcfd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6982b5bc2eec4c758d9321d73012474f in term 0.
I20260806 02:49:41.916421 30575 leader_election.cc:302] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6982b5bc2eec4c758d9321d73012474f, 81cee2d04b064bf68ffc64734307cbe6; no voters:
I20260806 02:49:41.916726 30961 raft_consensus.cc:2781] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:49:41.916824 30961 raft_consensus.cc:482] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:49:41.916870 30961 raft_consensus.cc:3037] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:41.917937 30961 raft_consensus.cc:504] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6982b5bc2eec4c758d9321d73012474f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34193 } } peers { permanent_uuid: "81cee2d04b064bf68ffc64734307cbe6" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 41609 } } peers { permanent_uuid: "ed6ef968f3ee4a8cb2aa60e0a97dcfd7" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37341 } }
I20260806 02:49:41.918309 30961 leader_election.cc:288] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f [CANDIDATE]: Term 1 election: Requested vote from peers 81cee2d04b064bf68ffc64734307cbe6 (127.1.183.129:41609), ed6ef968f3ee4a8cb2aa60e0a97dcfd7 (127.1.183.131:37341)
I20260806 02:49:41.918649 30479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b06a11565be84c10a9a4962223f45e48" candidate_uuid: "6982b5bc2eec4c758d9321d73012474f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81cee2d04b064bf68ffc64734307cbe6"
I20260806 02:49:41.918736 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b06a11565be84c10a9a4962223f45e48" candidate_uuid: "6982b5bc2eec4c758d9321d73012474f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed6ef968f3ee4a8cb2aa60e0a97dcfd7"
I20260806 02:49:41.918792 30479 raft_consensus.cc:3037] T b06a11565be84c10a9a4962223f45e48 P 81cee2d04b064bf68ffc64734307cbe6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:41.918882 30863 raft_consensus.cc:3037] T b06a11565be84c10a9a4962223f45e48 P ed6ef968f3ee4a8cb2aa60e0a97dcfd7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:41.919930 30479 raft_consensus.cc:2454] T b06a11565be84c10a9a4962223f45e48 P 81cee2d04b064bf68ffc64734307cbe6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6982b5bc2eec4c758d9321d73012474f in term 1.
I20260806 02:49:41.919958 30863 raft_consensus.cc:2454] T b06a11565be84c10a9a4962223f45e48 P ed6ef968f3ee4a8cb2aa60e0a97dcfd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6982b5bc2eec4c758d9321d73012474f in term 1.
I20260806 02:49:41.920305 30576 leader_election.cc:302] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6982b5bc2eec4c758d9321d73012474f, ed6ef968f3ee4a8cb2aa60e0a97dcfd7; no voters:
I20260806 02:49:41.920526 30961 raft_consensus.cc:2781] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:49:41.920784 30961 raft_consensus.cc:686] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f [term 1 LEADER]: Becoming Leader. State: Replica: 6982b5bc2eec4c758d9321d73012474f, State: Running, Role: LEADER
I20260806 02:49:41.921141 30961 consensus_queue.cc:237] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6982b5bc2eec4c758d9321d73012474f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34193 } } peers { permanent_uuid: "81cee2d04b064bf68ffc64734307cbe6" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 41609 } } peers { permanent_uuid: "ed6ef968f3ee4a8cb2aa60e0a97dcfd7" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37341 } }
I20260806 02:49:41.924543 30259 catalog_manager.cc:5495] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6982b5bc2eec4c758d9321d73012474f (127.1.183.130). New cstate: current_term: 1 leader_uuid: "6982b5bc2eec4c758d9321d73012474f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6982b5bc2eec4c758d9321d73012474f" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 34193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81cee2d04b064bf68ffc64734307cbe6" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 41609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed6ef968f3ee4a8cb2aa60e0a97dcfd7" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37341 } health_report { overall_health: UNKNOWN } } }
I20260806 02:49:41.942777 30947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:41.939356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42734 (local address 127.1.183.130:34193)
0806 02:49:41.939492 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:41.939495 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:41.939508 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:49:41.939618 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:41.939620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:41.939673 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:41.939760 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:49:41.940008 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.940571 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.941416 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:41.941617 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:41.941695 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:41.941999 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:49:41.942399 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:49:41.942490 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:41.942555 (+ 65us) server_negotiation.cc:299] Negotiation successful
0806 02:49:41.942604 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:49:41.948724 30479 raft_consensus.cc:1261] T b06a11565be84c10a9a4962223f45e48 P 81cee2d04b064bf68ffc64734307cbe6 [term 1 FOLLOWER]: Refusing update from remote peer 6982b5bc2eec4c758d9321d73012474f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:49:41.948731 30863 raft_consensus.cc:1261] T b06a11565be84c10a9a4962223f45e48 P ed6ef968f3ee4a8cb2aa60e0a97dcfd7 [term 1 FOLLOWER]: Refusing update from remote peer 6982b5bc2eec4c758d9321d73012474f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:49:41.949441 30969 consensus_queue.cc:1035] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f [LEADER]: Connected to new peer: Peer: permanent_uuid: "81cee2d04b064bf68ffc64734307cbe6" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 41609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:49:41.949674 30961 consensus_queue.cc:1035] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed6ef968f3ee4a8cb2aa60e0a97dcfd7" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 37341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:49:41.956422 30973 mvcc.cc:205] Tried to move back new op lower bound from 7315392847654105088 to 7315392847552069632. Current Snapshot: MvccSnapshot[applied={T|T < 7315392847654105088}]
I20260806 02:49:41.958387 30975 mvcc.cc:205] Tried to move back new op lower bound from 7315392847654105088 to 7315392847552069632. Current Snapshot: MvccSnapshot[applied={T|T < 7315392847654105088}]
I20260806 02:49:41.959709 30974 mvcc.cc:205] Tried to move back new op lower bound from 7315392847654105088 to 7315392847552069632. Current Snapshot: MvccSnapshot[applied={T|T < 7315392847654105088 or (T in {7315392847654105088})}]
I20260806 02:49:41.966384 30259 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:49:41.966574 30259 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:49:41.969178 30259 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 753e44c6be1f4ea689c82ebeb2d0db11: Sending DeleteTablet for 3 replicas of tablet b06a11565be84c10a9a4962223f45e48
I20260806 02:49:41.969982 30641 tablet_service.cc:1514] Processing DeleteTablet for tablet b06a11565be84c10a9a4962223f45e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:49:41 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:42726
I20260806 02:49:41.970213 30834 tablet_service.cc:1514] Processing DeleteTablet for tablet b06a11565be84c10a9a4962223f45e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:49:41 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:47098
I20260806 02:49:41.970587 30985 tablet_replica.cc:330] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f: stopping tablet replica
I20260806 02:49:41.970721 30447 tablet_service.cc:1514] Processing DeleteTablet for tablet b06a11565be84c10a9a4962223f45e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:49:41 UTC) from {username='kudu', principal='kudu/127.1.183.190@KRBTEST.COM'} at 127.0.0.1:34692
I20260806 02:49:41.971009 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 30353
I20260806 02:49:41.970991 30985 raft_consensus.cc:2229] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:49:41.971362 30985 raft_consensus.cc:2258] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:49:41.974893 30985 ts_tablet_manager.cc:1905] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:49:41.978534 30985 ts_tablet_manager.cc:1918] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:49:41.978685 30985 log.cc:1196] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-1/wal/wals/b06a11565be84c10a9a4962223f45e48
I20260806 02:49:41.979068 30985 ts_tablet_manager.cc:1939] T b06a11565be84c10a9a4962223f45e48 P 6982b5bc2eec4c758d9321d73012474f: Deleting consensus metadata
I20260806 02:49:41.979801 30987 tablet_replica.cc:330] T b06a11565be84c10a9a4962223f45e48 P ed6ef968f3ee4a8cb2aa60e0a97dcfd7: stopping tablet replica
I20260806 02:49:41.980348 30987 raft_consensus.cc:2229] T b06a11565be84c10a9a4962223f45e48 P ed6ef968f3ee4a8cb2aa60e0a97dcfd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:49:41.980566 30243 catalog_manager.cc:4841] TS 6982b5bc2eec4c758d9321d73012474f (127.1.183.130:34193): tablet b06a11565be84c10a9a4962223f45e48 (table test-table [id=2cc2a5567b5743e6a8bb7f6692b01448]) successfully deleted
I20260806 02:49:41.980934 30987 raft_consensus.cc:2258] T b06a11565be84c10a9a4962223f45e48 P ed6ef968f3ee4a8cb2aa60e0a97dcfd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:49:41.983777 30987 ts_tablet_manager.cc:1905] T b06a11565be84c10a9a4962223f45e48 P ed6ef968f3ee4a8cb2aa60e0a97dcfd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:49:41.987627 30987 ts_tablet_manager.cc:1918] T b06a11565be84c10a9a4962223f45e48 P ed6ef968f3ee4a8cb2aa60e0a97dcfd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:49:41.987787 30987 log.cc:1196] T b06a11565be84c10a9a4962223f45e48 P ed6ef968f3ee4a8cb2aa60e0a97dcfd7: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984313887056-1758-0/minicluster-data/ts-2/wal/wals/b06a11565be84c10a9a4962223f45e48
I20260806 02:49:41.988204 30987 ts_tablet_manager.cc:1939] T b06a11565be84c10a9a4962223f45e48 P ed6ef968f3ee4a8cb2aa60e0a97dcfd7: Deleting consensus metadata
I20260806 02:49:41.989647 30241 catalog_manager.cc:4841] TS ed6ef968f3ee4a8cb2aa60e0a97dcfd7 (127.1.183.131:37341): tablet b06a11565be84c10a9a4962223f45e48 (table test-table [id=2cc2a5567b5743e6a8bb7f6692b01448]) successfully deleted
W20260806 02:49:41.990015 30239 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.183.129:41609 (error 108)
I20260806 02:49:41.990213 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 30548
I20260806 02:49:41.996064 30948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:41.995733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:41609 (local address 127.0.0.1:34698)
0806 02:49:41.995852 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:41.995937 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.129:41609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260806 02:49:41.996299 30239 catalog_manager.cc:4584] TS 81cee2d04b064bf68ffc64734307cbe6 (127.1.183.129:41609): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b06a11565be84c10a9a4962223f45e48: Network error: Client connection negotiation failed: client connection to 127.1.183.129:41609: connect: Connection refused (error 111)
I20260806 02:49:42.003630 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 30739
I20260806 02:49:42.021979 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 30213
2026-08-06T02:49:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2909 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-06T02:49:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:49:42Z Disabled control of system clock
I20260806 02:49:42.062700 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:46213
--webserver_interface=127.1.183.190
--webserver_port=0
--builtin_ntp_servers=127.1.183.148:45611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:46213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:49:42.197444 31000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:42.197736 31000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:42.197774 31000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:42.202024 31000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:49:42.202093 31000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:42.202111 31000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:42.202127 31000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:49:42.202143 31000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:49:42.206925 31000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:45611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:46213
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:46213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:42.208030 31000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:42.209517 31000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:42.218173 31006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:42.218161 31012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:42.218161 31007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:42.218559 31000 server_base.cc:1034] running on GCE node
I20260806 02:49:42.219174 31000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:42.220378 31000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:42.221565 31000 hybrid_clock.cc:648] HybridClock initialized: now 1785984582221533 us; error 42 us; skew 500 ppm
I20260806 02:49:42.223438 31000 webserver.cc:460] Webserver started at http://127.1.183.190:33649/ using document root <none> and password file <none>
I20260806 02:49:42.224198 31000 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:42.224314 31000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:42.224627 31000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:42.226948 31000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "ba37babe80c64690b226f031c730942a"
format_stamp: "Formatted at 2026-08-06 02:49:42 on dist-test-slave-gmjp"
server_key: "bbf2a68f9f147a51ca9fe2cb124222be"
server_key_iv: "2f6bc73bf21b40755be32a7bae7c59c0"
server_key_version: "encryptionkey@0"
I20260806 02:49:42.227555 31000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "ba37babe80c64690b226f031c730942a"
format_stamp: "Formatted at 2026-08-06 02:49:42 on dist-test-slave-gmjp"
server_key: "bbf2a68f9f147a51ca9fe2cb124222be"
server_key_iv: "2f6bc73bf21b40755be32a7bae7c59c0"
server_key_version: "encryptionkey@0"
I20260806 02:49:42.231331 31000 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 02:49:42.233819 31019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:42.235083 31000 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:49:42.235230 31000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "ba37babe80c64690b226f031c730942a"
format_stamp: "Formatted at 2026-08-06 02:49:42 on dist-test-slave-gmjp"
server_key: "bbf2a68f9f147a51ca9fe2cb124222be"
server_key_iv: "2f6bc73bf21b40755be32a7bae7c59c0"
server_key_version: "encryptionkey@0"
I20260806 02:49:42.235345 31000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:42.249879 31000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:42.250624 31000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:42.250844 31000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:42.258293 31000 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:46213
I20260806 02:49:42.258316 31086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:46213 every 8 connection(s)
I20260806 02:49:42.259341 31000 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:49:42.261516 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 31000
I20260806 02:49:42.261660 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:49:42.261984 1758 external_mini_cluster.cc:1428] Setting key 91d88ca5b53e507be0b5c8e138680894
I20260806 02:49:42.263064 31087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:42.269394 31087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba37babe80c64690b226f031c730942a: Bootstrap starting.
I20260806 02:49:42.271888 31087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba37babe80c64690b226f031c730942a: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:42.271976 31090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:42.263542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32886 (local address 127.1.183.190:46213)
0806 02:49:42.264335 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:42.264360 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:42.264409 (+ 49us) server_negotiation.cc:407] Connection header received
0806 02:49:42.265098 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:42.265115 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:42.265516 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:42.265877 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:42.266311 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:42.267287 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:42.268220 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:42.268698 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:42.268889 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:42.270083 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:42.270095 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:42.270109 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:49:42.270539 (+ 430us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:42.270621 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:49:42.270642 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:42.270779 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:42.270931 (+ 152us) server_negotiation.cc:299] Negotiation successful
0806 02:49:42.271225 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":129,"threads_started":1}
I20260806 02:49:42.272763 31087 log.cc:824] T 00000000000000000000000000000000 P ba37babe80c64690b226f031c730942a: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:42.275010 31087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba37babe80c64690b226f031c730942a: No bootstrap required, opened a new log
I20260806 02:49:42.279078 31087 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba37babe80c64690b226f031c730942a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba37babe80c64690b226f031c730942a" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 46213 } }
I20260806 02:49:42.279346 31087 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba37babe80c64690b226f031c730942a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:42.279430 31087 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba37babe80c64690b226f031c730942a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba37babe80c64690b226f031c730942a, State: Initialized, Role: FOLLOWER
I20260806 02:49:42.279913 31087 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba37babe80c64690b226f031c730942a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba37babe80c64690b226f031c730942a" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 46213 } }
I20260806 02:49:42.280066 31087 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba37babe80c64690b226f031c730942a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:49:42.280153 31087 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba37babe80c64690b226f031c730942a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:49:42.280310 31087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba37babe80c64690b226f031c730942a [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:42.281541 31087 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba37babe80c64690b226f031c730942a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba37babe80c64690b226f031c730942a" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 46213 } }
I20260806 02:49:42.281921 31087 leader_election.cc:302] T 00000000000000000000000000000000 P ba37babe80c64690b226f031c730942a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba37babe80c64690b226f031c730942a; no voters:
I20260806 02:49:42.282210 31087 leader_election.cc:288] T 00000000000000000000000000000000 P ba37babe80c64690b226f031c730942a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:49:42.282343 31095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba37babe80c64690b226f031c730942a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:49:42.282617 31095 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba37babe80c64690b226f031c730942a [term 1 LEADER]: Becoming Leader. State: Replica: ba37babe80c64690b226f031c730942a, State: Running, Role: LEADER
I20260806 02:49:42.283026 31095 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba37babe80c64690b226f031c730942a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba37babe80c64690b226f031c730942a" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 46213 } }
I20260806 02:49:42.283646 31087 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba37babe80c64690b226f031c730942a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:49:42.284878 31098 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba37babe80c64690b226f031c730942a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba37babe80c64690b226f031c730942a. Latest consensus state: current_term: 1 leader_uuid: "ba37babe80c64690b226f031c730942a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba37babe80c64690b226f031c730942a" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 46213 } } }
I20260806 02:49:42.284883 31097 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba37babe80c64690b226f031c730942a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba37babe80c64690b226f031c730942a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba37babe80c64690b226f031c730942a" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 46213 } } }
I20260806 02:49:42.285033 31098 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba37babe80c64690b226f031c730942a [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:42.285033 31097 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba37babe80c64690b226f031c730942a [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:42.285494 31106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:49:42.288987 31106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:49:42.295768 31106 catalog_manager.cc:1347] Generated new cluster ID: 1d23610a4d2244108fe4025985335cdb
I20260806 02:49:42.295864 31106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:49:42.311515 31106 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:49:42.312556 31106 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:49:42.318626 31106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba37babe80c64690b226f031c730942a: Generated new TSK 0
I20260806 02:49:42.319357 31106 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:49:42.331954 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.129:0
--local_ip_for_outbound_sockets=127.1.183.129
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:46213
--builtin_ntp_servers=127.1.183.148:45611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:49:42.449494 31127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:42.449731 31127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:42.449771 31127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:42.453889 31127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:42.453943 31127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:42.453995 31127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.129
I20260806 02:49:42.459355 31127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:45611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.183.129
--webserver_port=0
--tserver_master_addrs=127.1.183.190:46213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.129
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:42.460438 31127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:42.461771 31127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:42.471199 31134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:42.471215 31135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:42.471210 31137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:42.471498 31127 server_base.cc:1034] running on GCE node
I20260806 02:49:42.471985 31127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:42.472679 31127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:42.473869 31127 hybrid_clock.cc:648] HybridClock initialized: now 1785984582473849 us; error 29 us; skew 500 ppm
I20260806 02:49:42.475603 31127 webserver.cc:460] Webserver started at http://127.1.183.129:40301/ using document root <none> and password file <none>
I20260806 02:49:42.476241 31127 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:42.476334 31127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:42.476562 31127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:42.478634 31127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-0/data/instance:
uuid: "85326676cc7d4073998b7348e86c58f6"
format_stamp: "Formatted at 2026-08-06 02:49:42 on dist-test-slave-gmjp"
server_key: "36060d899255a07fbd160b629cc857d7"
server_key_iv: "a7498f0600db427cc9ba2607dcf5e46a"
server_key_version: "encryptionkey@0"
I20260806 02:49:42.479190 31127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance:
uuid: "85326676cc7d4073998b7348e86c58f6"
format_stamp: "Formatted at 2026-08-06 02:49:42 on dist-test-slave-gmjp"
server_key: "36060d899255a07fbd160b629cc857d7"
server_key_iv: "a7498f0600db427cc9ba2607dcf5e46a"
server_key_version: "encryptionkey@0"
I20260806 02:49:42.483119 31127 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 02:49:42.485737 31147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:42.486968 31127 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:49:42.487125 31127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-0/wal
uuid: "85326676cc7d4073998b7348e86c58f6"
format_stamp: "Formatted at 2026-08-06 02:49:42 on dist-test-slave-gmjp"
server_key: "36060d899255a07fbd160b629cc857d7"
server_key_iv: "a7498f0600db427cc9ba2607dcf5e46a"
server_key_version: "encryptionkey@0"
I20260806 02:49:42.487290 31127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:42.496942 31127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:42.497570 31127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:42.497762 31127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:42.498328 31127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:49:42.499264 31127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:49:42.499336 31127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:42.499402 31127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:49:42.499446 31127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:42.508286 31127 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.129:40615
I20260806 02:49:42.508311 31311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.129:40615 every 8 connection(s)
I20260806 02:49:42.509202 31127 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-0/data/info.pb
I20260806 02:49:42.509626 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 31127
I20260806 02:49:42.509783 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-0/wal/instance
I20260806 02:49:42.510130 1758 external_mini_cluster.cc:1428] Setting key 1c2c27a3b87f8a55973c2148b6e27dfd
I20260806 02:49:42.512337 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.130:0
--local_ip_for_outbound_sockets=127.1.183.130
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:46213
--builtin_ntp_servers=127.1.183.148:45611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:49:42.520319 31090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:42.511113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.129:40817 (local address 127.1.183.190:46213)
0806 02:49:42.511260 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:42.511265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:42.512653 (+ 1388us) server_negotiation.cc:407] Connection header received
0806 02:49:42.514129 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:42.514131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:42.514185 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:42.514258 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:42.515537 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:42.516068 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:42.518026 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:42.518253 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:42.518335 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:42.519768 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:42.519771 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:42.519774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:49:42.519787 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:42.519805 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:49:42.519812 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:42.519877 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:42.520074 (+ 197us) server_negotiation.cc:299] Negotiation successful
0806 02:49:42.520136 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:49:42.521641 31315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:42.511689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46213 (local address 127.1.183.129:40817)
0806 02:49:42.512489 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:42.512543 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:42.513839 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:42.514493 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:42.514505 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:42.514625 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:49:42.515362 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:42.515401 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:42.516240 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:42.516247 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:42.517894 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:42.517910 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:42.518443 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:42.518460 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:42.518717 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:42.519425 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:49:42.519429 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:49:42.519495 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:49:42.519597 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:49:42.519906 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:49:42.519918 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:49:42.519927 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 02:49:42.520080 (+ 153us) client_negotiation.cc:240] Negotiation successful
0806 02:49:42.520458 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":153,"threads_started":1}
I20260806 02:49:42.523027 31313 heartbeater.cc:347] Connected to a master server at 127.1.183.190:46213
I20260806 02:49:42.523273 31313 heartbeater.cc:464] Registering TS with master...
I20260806 02:49:42.523813 31313 heartbeater.cc:511] Master 127.1.183.190:46213 requested a full tablet report, sending...
I20260806 02:49:42.525424 31038 ts_manager.cc:194] Registered new tserver with Master: 85326676cc7d4073998b7348e86c58f6 (127.1.183.129:40615)
I20260806 02:49:42.526576 31038 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.183.129:40817
W20260806 02:49:42.629842 31316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:42.630116 31316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:42.630155 31316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:42.634255 31316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:42.634318 31316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:42.634373 31316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.130
I20260806 02:49:42.639178 31316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:45611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.183.130
--webserver_port=0
--tserver_master_addrs=127.1.183.190:46213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.130
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:42.640331 31316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:42.641738 31316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:42.651127 31325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:42.651257 31323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:42.651371 31316 server_base.cc:1034] running on GCE node
W20260806 02:49:42.651132 31322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:42.651919 31316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:42.652653 31316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:42.653841 31316 hybrid_clock.cc:648] HybridClock initialized: now 1785984582653804 us; error 47 us; skew 500 ppm
I20260806 02:49:42.655645 31316 webserver.cc:460] Webserver started at http://127.1.183.130:36615/ using document root <none> and password file <none>
I20260806 02:49:42.656327 31316 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:42.656437 31316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:42.656666 31316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:42.658787 31316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-1/data/instance:
uuid: "30a0e967f3d8485bbd5fd92276c55241"
format_stamp: "Formatted at 2026-08-06 02:49:42 on dist-test-slave-gmjp"
server_key: "15528a95a2671f708952fccc9b380427"
server_key_iv: "c637aacd63d9d4efd40355d93cc3f2f3"
server_key_version: "encryptionkey@0"
I20260806 02:49:42.659441 31316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance:
uuid: "30a0e967f3d8485bbd5fd92276c55241"
format_stamp: "Formatted at 2026-08-06 02:49:42 on dist-test-slave-gmjp"
server_key: "15528a95a2671f708952fccc9b380427"
server_key_iv: "c637aacd63d9d4efd40355d93cc3f2f3"
server_key_version: "encryptionkey@0"
I20260806 02:49:42.663224 31316 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:49:42.665869 31333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:42.667109 31316 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:49:42.667258 31316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-1/wal
uuid: "30a0e967f3d8485bbd5fd92276c55241"
format_stamp: "Formatted at 2026-08-06 02:49:42 on dist-test-slave-gmjp"
server_key: "15528a95a2671f708952fccc9b380427"
server_key_iv: "c637aacd63d9d4efd40355d93cc3f2f3"
server_key_version: "encryptionkey@0"
I20260806 02:49:42.667383 31316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:42.676946 31316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:42.677611 31316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:42.677810 31316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:42.678412 31316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:49:42.679378 31316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:49:42.679457 31316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:42.679531 31316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:49:42.679572 31316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:42.689280 31316 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.130:37499
I20260806 02:49:42.689316 31491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.130:37499 every 8 connection(s)
I20260806 02:49:42.690248 31316 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-1/data/info.pb
I20260806 02:49:42.699375 31090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:42.692214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:60041 (local address 127.1.183.190:46213)
0806 02:49:42.692378 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:42.692383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:42.693472 (+ 1089us) server_negotiation.cc:407] Connection header received
0806 02:49:42.694817 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:42.694820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:42.694866 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:42.694947 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:42.695797 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:42.696401 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:42.697527 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:42.697731 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:42.697806 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:42.698967 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:42.698970 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:42.698972 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:49:42.698984 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:42.699000 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:49:42.699007 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:42.699061 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:42.699205 (+ 144us) server_negotiation.cc:299] Negotiation successful
0806 02:49:42.699255 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 02:49:42.700089 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 31316
I20260806 02:49:42.700255 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-1/wal/instance
I20260806 02:49:42.700443 31494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:42.692663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46213 (local address 127.1.183.130:60041)
0806 02:49:42.693305 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:42.693346 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:42.694623 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:42.695090 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:42.695096 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:42.695180 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:49:42.695668 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:42.695681 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:42.696527 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:42.696529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:42.697419 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:42.697428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:42.697876 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:42.697879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:42.698017 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:42.698709 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:49:42.698711 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:49:42.698752 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:49:42.698816 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:49:42.699078 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:49:42.699085 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:49:42.699090 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 02:49:42.699185 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 02:49:42.699403 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":116,"threads_started":1}
I20260806 02:49:42.700577 1758 external_mini_cluster.cc:1428] Setting key 3f78a0bf884d355aa378d6e6b1122e0d
I20260806 02:49:42.701714 31492 heartbeater.cc:347] Connected to a master server at 127.1.183.190:46213
I20260806 02:49:42.701948 31492 heartbeater.cc:464] Registering TS with master...
I20260806 02:49:42.702471 31492 heartbeater.cc:511] Master 127.1.183.190:46213 requested a full tablet report, sending...
I20260806 02:49:42.702625 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.183.131:0
--local_ip_for_outbound_sockets=127.1.183.131
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:46213
--builtin_ntp_servers=127.1.183.148:45611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:49:42.703594 31038 ts_manager.cc:194] Registered new tserver with Master: 30a0e967f3d8485bbd5fd92276c55241 (127.1.183.130:37499)
I20260806 02:49:42.704226 31038 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.183.130:60041
W20260806 02:49:42.819825 31497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:42.820107 31497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:42.820197 31497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:42.824345 31497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:42.824411 31497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:42.824468 31497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.183.131
I20260806 02:49:42.829300 31497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:45611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.183.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.183.131
--webserver_port=0
--tserver_master_addrs=127.1.183.190:46213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.183.131
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:42.830470 31497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:42.831840 31497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:42.841583 31509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:42.841564 31504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:42.841699 31497 server_base.cc:1034] running on GCE node
W20260806 02:49:42.841562 31505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:42.842203 31497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:42.842880 31497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:42.844071 31497 hybrid_clock.cc:648] HybridClock initialized: now 1785984582844042 us; error 48 us; skew 500 ppm
I20260806 02:49:42.845834 31497 webserver.cc:460] Webserver started at http://127.1.183.131:43027/ using document root <none> and password file <none>
I20260806 02:49:42.846422 31497 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:42.846518 31497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:42.846753 31497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:42.848868 31497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-2/data/instance:
uuid: "aaec099b35bf4c899c3a21bb974c3e6a"
format_stamp: "Formatted at 2026-08-06 02:49:42 on dist-test-slave-gmjp"
server_key: "b6ed447124b2db8b3baa1079ccd8aab0"
server_key_iv: "503764acd22ab99fc1d89baff66e8f40"
server_key_version: "encryptionkey@0"
I20260806 02:49:42.849447 31497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance:
uuid: "aaec099b35bf4c899c3a21bb974c3e6a"
format_stamp: "Formatted at 2026-08-06 02:49:42 on dist-test-slave-gmjp"
server_key: "b6ed447124b2db8b3baa1079ccd8aab0"
server_key_iv: "503764acd22ab99fc1d89baff66e8f40"
server_key_version: "encryptionkey@0"
I20260806 02:49:42.853026 31497 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:49:42.855357 31516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:42.856631 31497 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:49:42.856768 31497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-2/wal
uuid: "aaec099b35bf4c899c3a21bb974c3e6a"
format_stamp: "Formatted at 2026-08-06 02:49:42 on dist-test-slave-gmjp"
server_key: "b6ed447124b2db8b3baa1079ccd8aab0"
server_key_iv: "503764acd22ab99fc1d89baff66e8f40"
server_key_version: "encryptionkey@0"
I20260806 02:49:42.856880 31497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:42.889675 31497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:42.890507 31497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:42.890769 31497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:42.891453 31497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:49:42.892522 31497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:49:42.892596 31497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:42.892670 31497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:49:42.892714 31497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:42.902052 31497 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.131:41735
I20260806 02:49:42.902091 31682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.131:41735 every 8 connection(s)
I20260806 02:49:42.902990 31497 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-2/data/info.pb
I20260806 02:49:42.911043 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 31497
I20260806 02:49:42.911144 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-2/wal/instance
I20260806 02:49:42.911480 1758 external_mini_cluster.cc:1428] Setting key 9cc76e5b0e98f1a111803a53e6f2809a
I20260806 02:49:42.912555 31090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:42.904903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.131:59621 (local address 127.1.183.190:46213)
0806 02:49:42.905048 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:42.905052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:42.906091 (+ 1039us) server_negotiation.cc:407] Connection header received
0806 02:49:42.907569 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:42.907573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:42.907637 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:42.907740 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:42.908718 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:42.909259 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:42.910439 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:42.910672 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:42.910787 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:42.911985 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:42.911988 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:42.911990 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:49:42.912004 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:42.912024 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:49:42.912032 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:42.912109 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:42.912347 (+ 238us) server_negotiation.cc:299] Negotiation successful
0806 02:49:42.912414 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:49:42.913625 31687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:42.905271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:46213 (local address 127.1.183.131:59621)
0806 02:49:42.905934 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:42.905977 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:42.907298 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:42.907921 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:42.907927 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:42.908011 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:49:42.908576 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:42.908590 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:42.909415 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:42.909418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:42.910305 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:42.910314 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:42.910834 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:42.910838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:42.911003 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:42.911713 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:49:42.911715 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:49:42.911755 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:49:42.911820 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:49:42.912124 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:49:42.912137 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:49:42.912148 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 02:49:42.912343 (+ 195us) client_negotiation.cc:240] Negotiation successful
0806 02:49:42.912657 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":126,"threads_started":1}
I20260806 02:49:42.914753 31684 heartbeater.cc:347] Connected to a master server at 127.1.183.190:46213
I20260806 02:49:42.915002 31684 heartbeater.cc:464] Registering TS with master...
I20260806 02:49:42.915592 31684 heartbeater.cc:511] Master 127.1.183.190:46213 requested a full tablet report, sending...
I20260806 02:49:42.916678 31038 ts_manager.cc:194] Registered new tserver with Master: aaec099b35bf4c899c3a21bb974c3e6a (127.1.183.131:41735)
I20260806 02:49:42.917263 31038 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.183.131:59621
I20260806 02:49:42.925737 1758 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:49:42.931512 31090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:42.927798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32888 (local address 127.1.183.190:46213)
0806 02:49:42.927970 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:42.927975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:42.928188 (+ 213us) server_negotiation.cc:407] Connection header received
0806 02:49:42.928318 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:42.928320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:42.928363 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:42.928430 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:42.928699 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:42.929210 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:42.929997 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:42.930200 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:42.930275 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:42.931094 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:42.931097 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:42.931099 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:49:42.931110 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:42.931124 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:49:42.931130 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:42.931182 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:42.931277 (+ 95us) server_negotiation.cc:299] Negotiation successful
0806 02:49:42.931324 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:49:42.934293 31038 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:49:42.937302 31038 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:49:42.951546 31702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:42.946569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:41735 (local address 127.0.0.1:39402)
0806 02:49:42.947157 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:42.947171 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:42.947257 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:42.947968 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:42.947970 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:42.947987 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:42.948063 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:42.948069 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:42.949004 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:42.949009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:42.950178 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:42.950187 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:42.950961 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:42.950964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:42.951070 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:42.951107 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 02:49:42.951262 (+ 155us) client_negotiation.cc:240] Negotiation successful
0806 02:49:42.951382 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":3,"thread_start_us":91,"threads_started":1}
I20260806 02:49:42.951556 31705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:42.947077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39402 (local address 127.1.183.131:41735)
0806 02:49:42.947533 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:42.947545 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:42.947567 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:49:42.947636 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:42.947641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:42.947825 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:42.947998 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:42.948143 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:42.948891 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:42.950299 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:42.950833 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:42.950961 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:42.951172 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:42.951288 (+ 116us) server_negotiation.cc:299] Negotiation successful
0806 02:49:42.951381 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":130,"threads_started":1}
I20260806 02:49:42.951871 31697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:42.945872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:40615 (local address 127.0.0.1:39842)
0806 02:49:42.946865 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:42.946880 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:42.946969 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:42.947650 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:42.947653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:42.947672 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:42.947782 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:42.947810 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:42.949326 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:42.949329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:42.950289 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:42.950300 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:42.951368 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:42.951372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:42.951573 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:42.951600 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 02:49:42.951661 (+ 61us) client_negotiation.cc:240] Negotiation successful
0806 02:49:42.951730 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":932,"mutex_wait_us":70,"thread_start_us":102,"threads_started":1}
I20260806 02:49:42.952613 31703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:42.946646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39842 (local address 127.1.183.129:40615)
0806 02:49:42.947221 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:42.947232 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:42.947257 (+ 25us) server_negotiation.cc:407] Connection header received
0806 02:49:42.947323 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:42.947327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:42.947495 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:42.947972 (+ 477us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:42.948110 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:42.949213 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:42.950422 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:42.951231 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:42.952041 (+ 810us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:42.952262 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:42.952314 (+ 52us) server_negotiation.cc:299] Negotiation successful
0806 02:49:42.952430 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":117,"threads_started":1}
I20260806 02:49:42.953141 31698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:42.945831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.130:37499 (local address 127.0.0.1:55812)
0806 02:49:42.946304 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:42.946355 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:42.946492 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:42.947551 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:42.947557 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:42.947583 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:42.947727 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:42.947733 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:42.949250 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:42.949256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:42.951564 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:42.951573 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:42.952689 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:42.952694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:42.952797 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:42.952881 (+ 84us) client_negotiation.cc:769] Sending connection context
0806 02:49:42.952943 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 02:49:42.953015 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":103,"thread_start_us":86,"threads_started":1}
I20260806 02:49:42.953835 31699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:42.945981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55812 (local address 127.1.183.130:37499)
0806 02:49:42.946870 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:42.946889 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:42.946917 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:49:42.947154 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:42.947161 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:42.947369 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:42.947547 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:42.947856 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:42.949082 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:42.951699 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:42.952538 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:42.953325 (+ 787us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:42.953510 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:42.953565 (+ 55us) server_negotiation.cc:299] Negotiation successful
0806 02:49:42.953651 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":777,"thread_start_us":141,"threads_started":1}
I20260806 02:49:42.955313 31586 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6a3f2044d84eb6bc7b13634aa02e96 (DEFAULT_TABLE table=test-table [id=983e2957dd1e4f2db2948967a13b75bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:49:42.955314 31213 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6a3f2044d84eb6bc7b13634aa02e96 (DEFAULT_TABLE table=test-table [id=983e2957dd1e4f2db2948967a13b75bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:49:42.956504 31586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6a3f2044d84eb6bc7b13634aa02e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:42.956504 31213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6a3f2044d84eb6bc7b13634aa02e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:42.956099 31403 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6a3f2044d84eb6bc7b13634aa02e96 (DEFAULT_TABLE table=test-table [id=983e2957dd1e4f2db2948967a13b75bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:49:42.957170 31403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6a3f2044d84eb6bc7b13634aa02e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:42.963333 31707 tablet_bootstrap.cc:492] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 30a0e967f3d8485bbd5fd92276c55241: Bootstrap starting.
I20260806 02:49:42.964468 31708 tablet_bootstrap.cc:492] T 3f6a3f2044d84eb6bc7b13634aa02e96 P aaec099b35bf4c899c3a21bb974c3e6a: Bootstrap starting.
I20260806 02:49:42.964870 31710 tablet_bootstrap.cc:492] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 85326676cc7d4073998b7348e86c58f6: Bootstrap starting.
I20260806 02:49:42.965862 31707 tablet_bootstrap.cc:654] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 30a0e967f3d8485bbd5fd92276c55241: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:42.966321 31708 tablet_bootstrap.cc:654] T 3f6a3f2044d84eb6bc7b13634aa02e96 P aaec099b35bf4c899c3a21bb974c3e6a: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:42.966787 31707 log.cc:824] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 30a0e967f3d8485bbd5fd92276c55241: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:42.967006 31710 tablet_bootstrap.cc:654] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 85326676cc7d4073998b7348e86c58f6: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:42.967144 31708 log.cc:824] T 3f6a3f2044d84eb6bc7b13634aa02e96 P aaec099b35bf4c899c3a21bb974c3e6a: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:42.967809 31710 log.cc:824] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 85326676cc7d4073998b7348e86c58f6: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:42.969142 31708 tablet_bootstrap.cc:492] T 3f6a3f2044d84eb6bc7b13634aa02e96 P aaec099b35bf4c899c3a21bb974c3e6a: No bootstrap required, opened a new log
I20260806 02:49:42.969334 31708 ts_tablet_manager.cc:1397] T 3f6a3f2044d84eb6bc7b13634aa02e96 P aaec099b35bf4c899c3a21bb974c3e6a: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260806 02:49:42.969717 31710 tablet_bootstrap.cc:492] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 85326676cc7d4073998b7348e86c58f6: No bootstrap required, opened a new log
I20260806 02:49:42.969916 31710 ts_tablet_manager.cc:1397] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 85326676cc7d4073998b7348e86c58f6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260806 02:49:42.971125 31707 tablet_bootstrap.cc:492] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 30a0e967f3d8485bbd5fd92276c55241: No bootstrap required, opened a new log
I20260806 02:49:42.971319 31707 ts_tablet_manager.cc:1397] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 30a0e967f3d8485bbd5fd92276c55241: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260806 02:49:42.973070 31708 raft_consensus.cc:348] T 3f6a3f2044d84eb6bc7b13634aa02e96 P aaec099b35bf4c899c3a21bb974c3e6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a0e967f3d8485bbd5fd92276c55241" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37499 } } peers { permanent_uuid: "85326676cc7d4073998b7348e86c58f6" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40615 } } peers { permanent_uuid: "aaec099b35bf4c899c3a21bb974c3e6a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 41735 } }
I20260806 02:49:42.973308 31708 raft_consensus.cc:374] T 3f6a3f2044d84eb6bc7b13634aa02e96 P aaec099b35bf4c899c3a21bb974c3e6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:42.973362 31708 raft_consensus.cc:729] T 3f6a3f2044d84eb6bc7b13634aa02e96 P aaec099b35bf4c899c3a21bb974c3e6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaec099b35bf4c899c3a21bb974c3e6a, State: Initialized, Role: FOLLOWER
I20260806 02:49:42.973888 31708 consensus_queue.cc:260] T 3f6a3f2044d84eb6bc7b13634aa02e96 P aaec099b35bf4c899c3a21bb974c3e6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a0e967f3d8485bbd5fd92276c55241" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37499 } } peers { permanent_uuid: "85326676cc7d4073998b7348e86c58f6" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40615 } } peers { permanent_uuid: "aaec099b35bf4c899c3a21bb974c3e6a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 41735 } }
I20260806 02:49:42.974325 31710 raft_consensus.cc:348] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 85326676cc7d4073998b7348e86c58f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a0e967f3d8485bbd5fd92276c55241" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37499 } } peers { permanent_uuid: "85326676cc7d4073998b7348e86c58f6" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40615 } } peers { permanent_uuid: "aaec099b35bf4c899c3a21bb974c3e6a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 41735 } }
I20260806 02:49:42.974587 31710 raft_consensus.cc:374] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 85326676cc7d4073998b7348e86c58f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:42.974648 31710 raft_consensus.cc:729] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 85326676cc7d4073998b7348e86c58f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85326676cc7d4073998b7348e86c58f6, State: Initialized, Role: FOLLOWER
I20260806 02:49:42.975054 31708 ts_tablet_manager.cc:1428] T 3f6a3f2044d84eb6bc7b13634aa02e96 P aaec099b35bf4c899c3a21bb974c3e6a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:49:42.975098 31710 consensus_queue.cc:260] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 85326676cc7d4073998b7348e86c58f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a0e967f3d8485bbd5fd92276c55241" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37499 } } peers { permanent_uuid: "85326676cc7d4073998b7348e86c58f6" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40615 } } peers { permanent_uuid: "aaec099b35bf4c899c3a21bb974c3e6a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 41735 } }
I20260806 02:49:42.975172 31707 raft_consensus.cc:348] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 30a0e967f3d8485bbd5fd92276c55241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a0e967f3d8485bbd5fd92276c55241" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37499 } } peers { permanent_uuid: "85326676cc7d4073998b7348e86c58f6" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40615 } } peers { permanent_uuid: "aaec099b35bf4c899c3a21bb974c3e6a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 41735 } }
I20260806 02:49:42.975472 31684 heartbeater.cc:503] Master 127.1.183.190:46213 was elected leader, sending a full tablet report...
I20260806 02:49:42.975507 31707 raft_consensus.cc:374] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 30a0e967f3d8485bbd5fd92276c55241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:42.975564 31707 raft_consensus.cc:729] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 30a0e967f3d8485bbd5fd92276c55241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a0e967f3d8485bbd5fd92276c55241, State: Initialized, Role: FOLLOWER
I20260806 02:49:42.975874 31313 heartbeater.cc:503] Master 127.1.183.190:46213 was elected leader, sending a full tablet report...
I20260806 02:49:42.975996 31707 consensus_queue.cc:260] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 30a0e967f3d8485bbd5fd92276c55241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a0e967f3d8485bbd5fd92276c55241" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37499 } } peers { permanent_uuid: "85326676cc7d4073998b7348e86c58f6" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40615 } } peers { permanent_uuid: "aaec099b35bf4c899c3a21bb974c3e6a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 41735 } }
I20260806 02:49:42.976099 31710 ts_tablet_manager.cc:1428] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 85326676cc7d4073998b7348e86c58f6: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260806 02:49:42.976960 31707 ts_tablet_manager.cc:1428] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 30a0e967f3d8485bbd5fd92276c55241: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 02:49:42.977308 31492 heartbeater.cc:503] Master 127.1.183.190:46213 was elected leader, sending a full tablet report...
W20260806 02:49:43.010663 31314 tablet.cc:2369] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 85326676cc7d4073998b7348e86c58f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:49:43.077512 31718 raft_consensus.cc:482] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 30a0e967f3d8485bbd5fd92276c55241 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:49:43.077780 31718 raft_consensus.cc:504] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 30a0e967f3d8485bbd5fd92276c55241 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a0e967f3d8485bbd5fd92276c55241" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37499 } } peers { permanent_uuid: "85326676cc7d4073998b7348e86c58f6" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40615 } } peers { permanent_uuid: "aaec099b35bf4c899c3a21bb974c3e6a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 41735 } }
I20260806 02:49:43.079061 31718 leader_election.cc:288] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 30a0e967f3d8485bbd5fd92276c55241 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85326676cc7d4073998b7348e86c58f6 (127.1.183.129:40615), aaec099b35bf4c899c3a21bb974c3e6a (127.1.183.131:41735)
I20260806 02:49:43.084306 31703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:43.079385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:38347 (local address 127.1.183.129:40615)
0806 02:49:43.079507 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:43.079511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:43.079657 (+ 146us) server_negotiation.cc:407] Connection header received
0806 02:49:43.079730 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:43.079733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:43.079806 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:43.079916 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:43.080319 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:43.081123 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:43.082953 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:43.083561 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:43.083692 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:43.083744 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:43.084064 (+ 320us) server_negotiation.cc:299] Negotiation successful
0806 02:49:43.084142 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:49:43.084306 31705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:43.079470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.183.130:51811 (local address 127.1.183.131:41735)
0806 02:49:43.079608 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:43.079615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:43.079828 (+ 213us) server_negotiation.cc:407] Connection header received
0806 02:49:43.079900 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:43.079906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:43.079950 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:43.080030 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:49:43.080371 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:43.081123 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:43.082965 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:43.083563 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:43.083692 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:43.083744 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:43.084064 (+ 320us) server_negotiation.cc:299] Negotiation successful
0806 02:49:43.084142 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:49:43.084385 31721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:43.079350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.131:41735 (local address 127.1.183.130:51811)
0806 02:49:43.079723 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:43.079741 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:43.079818 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:43.080046 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:43.080048 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:43.080064 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:43.080243 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:43.080254 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:43.081240 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:43.081243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:43.082798 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:43.082811 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:43.083705 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:43.083709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:43.083928 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:43.083945 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 02:49:43.084025 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 02:49:43.084202 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":304,"thread_start_us":134,"threads_started":1}
I20260806 02:49:43.084863 31494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:43.079271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:40615 (local address 127.1.183.130:38347)
0806 02:49:43.079490 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:43.079506 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:49:43.079619 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:49:43.079943 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:49:43.079945 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:49:43.079960 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:49:43.080175 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:43.080187 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:43.081240 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:43.081243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:43.082798 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:49:43.082811 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:49:43.083707 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:49:43.083709 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:49:43.083884 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:43.083944 (+ 60us) client_negotiation.cc:769] Sending connection context
0806 02:49:43.084025 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 02:49:43.084341 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":368}
I20260806 02:49:43.085290 31615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6a3f2044d84eb6bc7b13634aa02e96" candidate_uuid: "30a0e967f3d8485bbd5fd92276c55241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaec099b35bf4c899c3a21bb974c3e6a" is_pre_election: true
I20260806 02:49:43.085572 31245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6a3f2044d84eb6bc7b13634aa02e96" candidate_uuid: "30a0e967f3d8485bbd5fd92276c55241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85326676cc7d4073998b7348e86c58f6" is_pre_election: true
I20260806 02:49:43.085675 31615 raft_consensus.cc:2454] T 3f6a3f2044d84eb6bc7b13634aa02e96 P aaec099b35bf4c899c3a21bb974c3e6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30a0e967f3d8485bbd5fd92276c55241 in term 0.
I20260806 02:49:43.085870 31245 raft_consensus.cc:2454] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 85326676cc7d4073998b7348e86c58f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30a0e967f3d8485bbd5fd92276c55241 in term 0.
I20260806 02:49:43.086246 31340 leader_election.cc:302] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 30a0e967f3d8485bbd5fd92276c55241 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30a0e967f3d8485bbd5fd92276c55241, aaec099b35bf4c899c3a21bb974c3e6a; no voters:
I20260806 02:49:43.086504 31718 raft_consensus.cc:2781] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 30a0e967f3d8485bbd5fd92276c55241 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:49:43.086596 31718 raft_consensus.cc:482] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 30a0e967f3d8485bbd5fd92276c55241 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:49:43.086639 31718 raft_consensus.cc:3037] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 30a0e967f3d8485bbd5fd92276c55241 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:43.087704 31718 raft_consensus.cc:504] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 30a0e967f3d8485bbd5fd92276c55241 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a0e967f3d8485bbd5fd92276c55241" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37499 } } peers { permanent_uuid: "85326676cc7d4073998b7348e86c58f6" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40615 } } peers { permanent_uuid: "aaec099b35bf4c899c3a21bb974c3e6a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 41735 } }
I20260806 02:49:43.088064 31718 leader_election.cc:288] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 30a0e967f3d8485bbd5fd92276c55241 [CANDIDATE]: Term 1 election: Requested vote from peers 85326676cc7d4073998b7348e86c58f6 (127.1.183.129:40615), aaec099b35bf4c899c3a21bb974c3e6a (127.1.183.131:41735)
I20260806 02:49:43.088419 31245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6a3f2044d84eb6bc7b13634aa02e96" candidate_uuid: "30a0e967f3d8485bbd5fd92276c55241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85326676cc7d4073998b7348e86c58f6"
I20260806 02:49:43.088528 31615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6a3f2044d84eb6bc7b13634aa02e96" candidate_uuid: "30a0e967f3d8485bbd5fd92276c55241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaec099b35bf4c899c3a21bb974c3e6a"
I20260806 02:49:43.088553 31245 raft_consensus.cc:3037] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 85326676cc7d4073998b7348e86c58f6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:43.088697 31615 raft_consensus.cc:3037] T 3f6a3f2044d84eb6bc7b13634aa02e96 P aaec099b35bf4c899c3a21bb974c3e6a [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:43.089627 31245 raft_consensus.cc:2454] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 85326676cc7d4073998b7348e86c58f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30a0e967f3d8485bbd5fd92276c55241 in term 1.
I20260806 02:49:43.089802 31615 raft_consensus.cc:2454] T 3f6a3f2044d84eb6bc7b13634aa02e96 P aaec099b35bf4c899c3a21bb974c3e6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30a0e967f3d8485bbd5fd92276c55241 in term 1.
I20260806 02:49:43.089959 31334 leader_election.cc:302] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 30a0e967f3d8485bbd5fd92276c55241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30a0e967f3d8485bbd5fd92276c55241, 85326676cc7d4073998b7348e86c58f6; no voters:
I20260806 02:49:43.090152 31718 raft_consensus.cc:2781] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 30a0e967f3d8485bbd5fd92276c55241 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:49:43.090435 31718 raft_consensus.cc:686] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 30a0e967f3d8485bbd5fd92276c55241 [term 1 LEADER]: Becoming Leader. State: Replica: 30a0e967f3d8485bbd5fd92276c55241, State: Running, Role: LEADER
I20260806 02:49:43.090839 31718 consensus_queue.cc:237] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 30a0e967f3d8485bbd5fd92276c55241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a0e967f3d8485bbd5fd92276c55241" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37499 } } peers { permanent_uuid: "85326676cc7d4073998b7348e86c58f6" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40615 } } peers { permanent_uuid: "aaec099b35bf4c899c3a21bb974c3e6a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 41735 } }
I20260806 02:49:43.094233 31038 catalog_manager.cc:5495] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 30a0e967f3d8485bbd5fd92276c55241 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30a0e967f3d8485bbd5fd92276c55241 (127.1.183.130). New cstate: current_term: 1 leader_uuid: "30a0e967f3d8485bbd5fd92276c55241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a0e967f3d8485bbd5fd92276c55241" member_type: VOTER last_known_addr { host: "127.1.183.130" port: 37499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85326676cc7d4073998b7348e86c58f6" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaec099b35bf4c899c3a21bb974c3e6a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 02:49:43.141044 31699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:43.137125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55820 (local address 127.1.183.130:37499)
0806 02:49:43.137290 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:43.137295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:43.137312 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:49:43.137504 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:43.137511 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:43.137574 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:43.137700 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:49:43.137966 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:43.138488 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:43.139461 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:43.139692 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:43.139807 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:43.140133 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:49:43.140588 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:49:43.140733 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:43.140795 (+ 62us) server_negotiation.cc:299] Negotiation successful
0806 02:49:43.140853 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 02:49:43.147061 31245 raft_consensus.cc:1261] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 85326676cc7d4073998b7348e86c58f6 [term 1 FOLLOWER]: Refusing update from remote peer 30a0e967f3d8485bbd5fd92276c55241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:49:43.147063 31615 raft_consensus.cc:1261] T 3f6a3f2044d84eb6bc7b13634aa02e96 P aaec099b35bf4c899c3a21bb974c3e6a [term 1 FOLLOWER]: Refusing update from remote peer 30a0e967f3d8485bbd5fd92276c55241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:49:43.148005 31722 consensus_queue.cc:1035] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 30a0e967f3d8485bbd5fd92276c55241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85326676cc7d4073998b7348e86c58f6" member_type: VOTER last_known_addr { host: "127.1.183.129" port: 40615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:49:43.148223 31718 consensus_queue.cc:1035] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 30a0e967f3d8485bbd5fd92276c55241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaec099b35bf4c899c3a21bb974c3e6a" member_type: VOTER last_known_addr { host: "127.1.183.131" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 02:49:43.153932 31686 tablet.cc:2369] T 3f6a3f2044d84eb6bc7b13634aa02e96 P aaec099b35bf4c899c3a21bb974c3e6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:49:43.157248 31727 mvcc.cc:205] Tried to move back new op lower bound from 7315392852562767872 to 7315392852342677504. Current Snapshot: MvccSnapshot[applied={T|T < 7315392852562767872}]
I20260806 02:49:43.159894 31726 mvcc.cc:205] Tried to move back new op lower bound from 7315392852562767872 to 7315392852342677504. Current Snapshot: MvccSnapshot[applied={T|T < 7315392852562767872}]
I20260806 02:49:43.164933 31038 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:32888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:49:43.165133 31038 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:32888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:49:43.167735 31038 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ba37babe80c64690b226f031c730942a: Sending DeleteTablet for 3 replicas of tablet 3f6a3f2044d84eb6bc7b13634aa02e96
I20260806 02:49:43.168663 31213 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f6a3f2044d84eb6bc7b13634aa02e96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:49:43 UTC) from {username='slave'} at 127.0.0.1:39842
I20260806 02:49:43.168659 31403 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f6a3f2044d84eb6bc7b13634aa02e96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:49:43 UTC) from {username='slave'} at 127.0.0.1:55812
I20260806 02:49:43.169255 31739 tablet_replica.cc:330] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 85326676cc7d4073998b7348e86c58f6: stopping tablet replica
I20260806 02:49:43.169401 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 31127
I20260806 02:49:43.169574 31586 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f6a3f2044d84eb6bc7b13634aa02e96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:49:43 UTC) from {username='slave'} at 127.0.0.1:39402
I20260806 02:49:43.169675 31739 raft_consensus.cc:2229] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 85326676cc7d4073998b7348e86c58f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:49:43.170116 31737 tablet_replica.cc:330] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 30a0e967f3d8485bbd5fd92276c55241: stopping tablet replica
I20260806 02:49:43.170526 31737 raft_consensus.cc:2229] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 30a0e967f3d8485bbd5fd92276c55241 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:49:43.171054 31737 raft_consensus.cc:2258] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 30a0e967f3d8485bbd5fd92276c55241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:49:43.175016 31737 ts_tablet_manager.cc:1905] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 30a0e967f3d8485bbd5fd92276c55241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:49:43.178354 31737 ts_tablet_manager.cc:1918] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 30a0e967f3d8485bbd5fd92276c55241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:49:43.178467 31737 log.cc:1196] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 30a0e967f3d8485bbd5fd92276c55241: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-1/wal/wals/3f6a3f2044d84eb6bc7b13634aa02e96
I20260806 02:49:43.178786 31737 ts_tablet_manager.cc:1939] T 3f6a3f2044d84eb6bc7b13634aa02e96 P 30a0e967f3d8485bbd5fd92276c55241: Deleting consensus metadata
I20260806 02:49:43.180619 31024 catalog_manager.cc:4841] TS 30a0e967f3d8485bbd5fd92276c55241 (127.1.183.130:37499): tablet 3f6a3f2044d84eb6bc7b13634aa02e96 (table test-table [id=983e2957dd1e4f2db2948967a13b75bf]) successfully deleted
I20260806 02:49:43.183375 31740 tablet_replica.cc:330] T 3f6a3f2044d84eb6bc7b13634aa02e96 P aaec099b35bf4c899c3a21bb974c3e6a: stopping tablet replica
I20260806 02:49:43.183800 31740 raft_consensus.cc:2229] T 3f6a3f2044d84eb6bc7b13634aa02e96 P aaec099b35bf4c899c3a21bb974c3e6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:49:43.184267 31740 raft_consensus.cc:2258] T 3f6a3f2044d84eb6bc7b13634aa02e96 P aaec099b35bf4c899c3a21bb974c3e6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:49:43.188912 31740 ts_tablet_manager.cc:1905] T 3f6a3f2044d84eb6bc7b13634aa02e96 P aaec099b35bf4c899c3a21bb974c3e6a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 02:49:43.192091 31020 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.183.129:40615 (error 108)
I20260806 02:49:43.192668 31740 ts_tablet_manager.cc:1918] T 3f6a3f2044d84eb6bc7b13634aa02e96 P aaec099b35bf4c899c3a21bb974c3e6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:49:43.192775 31740 log.cc:1196] T 3f6a3f2044d84eb6bc7b13634aa02e96 P aaec099b35bf4c899c3a21bb974c3e6a: Deleting WAL directory at /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984313887056-1758-0/minicluster-data/ts-2/wal/wals/3f6a3f2044d84eb6bc7b13634aa02e96
I20260806 02:49:43.193097 31740 ts_tablet_manager.cc:1939] T 3f6a3f2044d84eb6bc7b13634aa02e96 P aaec099b35bf4c899c3a21bb974c3e6a: Deleting consensus metadata
I20260806 02:49:43.193517 31698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:43.193131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.129:40615 (local address 127.0.0.1:39854)
0806 02:49:43.193369 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0806 02:49:43.193438 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.129:40615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
W20260806 02:49:43.193718 31020 catalog_manager.cc:4584] TS 85326676cc7d4073998b7348e86c58f6 (127.1.183.129:40615): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3f6a3f2044d84eb6bc7b13634aa02e96: Network error: Client connection negotiation failed: client connection to 127.1.183.129:40615: connect: Connection refused (error 111)
I20260806 02:49:43.194121 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 31316
I20260806 02:49:43.201071 31024 catalog_manager.cc:4841] TS aaec099b35bf4c899c3a21bb974c3e6a (127.1.183.131:41735): tablet 3f6a3f2044d84eb6bc7b13634aa02e96 (table test-table [id=983e2957dd1e4f2db2948967a13b75bf]) successfully deleted
I20260806 02:49:43.208453 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 31497
I20260806 02:49:43.219748 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 31000
2026-08-06T02:49:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1193 ms)
[----------] 6 tests from EncryptionPolicyTest (11616 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:49:43 dist-test-slave-gmjp krb5kdc[31747](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:49:43 dist-test-slave-gmjp krb5kdc[31747](info): set up 2 sockets
Aug 06 02:49:43 dist-test-slave-gmjp krb5kdc[31747](info): commencing operation
krb5kdc: starting...
W20260806 02:49:44.450544 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.204s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:49:44 dist-test-slave-gmjp krb5kdc[31747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:49:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:49:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:49:44.579486 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36605
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:49:44.693149 31767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:44.693451 31767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:44.693539 31767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:44.697611 31767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:49:44.697698 31767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:44.697737 31767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:44.697777 31767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:49:44.697809 31767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:49:44.703168 31767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36605
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:44.704313 31767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:44.705374 31767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:44.713696 31775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:44.713737 31776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:44.713701 31780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:44.714165 31767 server_base.cc:1034] running on GCE node
I20260806 02:49:44.714670 31767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:44.715840 31767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:44.717003 31767 hybrid_clock.cc:648] HybridClock initialized: now 1785984584716963 us; error 52 us; skew 500 ppm
Aug 06 02:49:44 dist-test-slave-gmjp krb5kdc[31747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:44.719655 31767 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:49:44.720531 31767 webserver.cc:460] Webserver started at http://127.0.0.1:39647/ using document root <none> and password file <none>
I20260806 02:49:44.721045 31767 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:44.721100 31767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:44.721264 31767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:44.723234 31767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "0017064d1e9d4f299f33c65f4df5e024"
format_stamp: "Formatted at 2026-08-06 02:49:44 on dist-test-slave-gmjp"
server_key: "1bdee976d2a546eb2c6b3ff32f971a9c"
server_key_iv: "bec19eb33658643502c4f093d2d2bedc"
server_key_version: "encryptionkey@0"
I20260806 02:49:44.723749 31767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "0017064d1e9d4f299f33c65f4df5e024"
format_stamp: "Formatted at 2026-08-06 02:49:44 on dist-test-slave-gmjp"
server_key: "1bdee976d2a546eb2c6b3ff32f971a9c"
server_key_iv: "bec19eb33658643502c4f093d2d2bedc"
server_key_version: "encryptionkey@0"
I20260806 02:49:44.727452 31767 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 02:49:44.729897 31789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:44.731141 31767 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:49:44.731287 31767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "0017064d1e9d4f299f33c65f4df5e024"
format_stamp: "Formatted at 2026-08-06 02:49:44 on dist-test-slave-gmjp"
server_key: "1bdee976d2a546eb2c6b3ff32f971a9c"
server_key_iv: "bec19eb33658643502c4f093d2d2bedc"
server_key_version: "encryptionkey@0"
I20260806 02:49:44.731405 31767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:44.759130 31767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:44.762161 31767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:44.762384 31767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:44.769817 31767 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36605
I20260806 02:49:44.769843 31864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36605 every 8 connection(s)
I20260806 02:49:44.770793 31767 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:49:44.773687 31865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:44.776353 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 31767
I20260806 02:49:44.776505 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:49:44.776809 1758 external_mini_cluster.cc:1428] Setting key 31f4c35cf88f6cc1064115d905bd30b6
I20260806 02:49:44.780107 31865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0017064d1e9d4f299f33c65f4df5e024: Bootstrap starting.
I20260806 02:49:44.782568 31865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0017064d1e9d4f299f33c65f4df5e024: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:44.783361 31865 log.cc:824] T 00000000000000000000000000000000 P 0017064d1e9d4f299f33c65f4df5e024: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:44.785586 31865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0017064d1e9d4f299f33c65f4df5e024: No bootstrap required, opened a new log
Aug 06 02:49:44 dist-test-slave-gmjp krb5kdc[31747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 02:49:44.789137 31865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0017064d1e9d4f299f33c65f4df5e024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0017064d1e9d4f299f33c65f4df5e024" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36605 } }
I20260806 02:49:44.789402 31865 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0017064d1e9d4f299f33c65f4df5e024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:44.789497 31865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0017064d1e9d4f299f33c65f4df5e024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0017064d1e9d4f299f33c65f4df5e024, State: Initialized, Role: FOLLOWER
I20260806 02:49:44.790022 31865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0017064d1e9d4f299f33c65f4df5e024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0017064d1e9d4f299f33c65f4df5e024" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36605 } }
I20260806 02:49:44.790239 31865 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0017064d1e9d4f299f33c65f4df5e024 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:49:44.790313 31865 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0017064d1e9d4f299f33c65f4df5e024 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:49:44.790403 31865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0017064d1e9d4f299f33c65f4df5e024 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:44.791652 31865 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0017064d1e9d4f299f33c65f4df5e024 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0017064d1e9d4f299f33c65f4df5e024" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36605 } }
I20260806 02:49:44.792018 31865 leader_election.cc:302] T 00000000000000000000000000000000 P 0017064d1e9d4f299f33c65f4df5e024 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0017064d1e9d4f299f33c65f4df5e024; no voters:
I20260806 02:49:44.792430 31865 leader_election.cc:288] T 00000000000000000000000000000000 P 0017064d1e9d4f299f33c65f4df5e024 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:49:44.792516 31872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:44.778545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42375 (local address 127.0.0.1:36605)
0806 02:49:44.778972 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:44.778998 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:44.779036 (+ 38us) server_negotiation.cc:407] Connection header received
0806 02:49:44.779659 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:44.779676 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:44.780035 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:44.780472 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:44.781324 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:44.782353 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:44.783300 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:44.783845 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:44.784028 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:44.786683 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:44.786688 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:44.786702 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:44.786731 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:44.789005 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:44.789597 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:44.789604 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:44.789609 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:44.789766 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:44.790120 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:44.790124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:44.790126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:44.790685 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:44.790916 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:44.791224 (+ 308us) server_negotiation.cc:299] Negotiation successful
0806 02:49:44.791475 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":131,"threads_started":1}
I20260806 02:49:44.792551 31878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0017064d1e9d4f299f33c65f4df5e024 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:49:44.792809 31878 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0017064d1e9d4f299f33c65f4df5e024 [term 1 LEADER]: Becoming Leader. State: Replica: 0017064d1e9d4f299f33c65f4df5e024, State: Running, Role: LEADER
I20260806 02:49:44.793128 31878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0017064d1e9d4f299f33c65f4df5e024 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0017064d1e9d4f299f33c65f4df5e024" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36605 } }
I20260806 02:49:44.793761 31865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0017064d1e9d4f299f33c65f4df5e024 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:49:44.794406 31879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0017064d1e9d4f299f33c65f4df5e024 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0017064d1e9d4f299f33c65f4df5e024. Latest consensus state: current_term: 1 leader_uuid: "0017064d1e9d4f299f33c65f4df5e024" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0017064d1e9d4f299f33c65f4df5e024" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36605 } } }
I20260806 02:49:44.794633 31879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0017064d1e9d4f299f33c65f4df5e024 [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:44.794919 31880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0017064d1e9d4f299f33c65f4df5e024 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0017064d1e9d4f299f33c65f4df5e024" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0017064d1e9d4f299f33c65f4df5e024" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36605 } } }
I20260806 02:49:44.795027 31880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0017064d1e9d4f299f33c65f4df5e024 [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:44.795445 31888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:49:44.796268 1758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:49:44.798789 31888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:49:44.804970 31888 catalog_manager.cc:1347] Generated new cluster ID: ac0061bc14cd4403a01151b08d582a0f
I20260806 02:49:44.805040 31888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:49:44.807185 31872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:44.798530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56133 (local address 127.0.0.1:36605)
0806 02:49:44.798643 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:44.798647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:44.798902 (+ 255us) server_negotiation.cc:407] Connection header received
0806 02:49:44.799152 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:44.799155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:44.799204 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:44.799256 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:44.800266 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:44.800805 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:44.801654 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:44.801903 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:44.802026 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:44.803657 (+ 1631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:44.803661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:44.803663 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:44.803696 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:44.805383 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:44.805865 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:44.805876 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:44.805878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:44.805959 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:44.806298 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:44.806301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:44.806303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:44.806481 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:44.806651 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:44.806935 (+ 284us) server_negotiation.cc:299] Negotiation successful
0806 02:49:44.807032 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 02:49:44.831462 31888 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:49:44.832803 31888 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:49:44.838737 31888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0017064d1e9d4f299f33c65f4df5e024: Generated new TSK 0
I20260806 02:49:44.839424 31888 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:49:44.845359 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 31767
W20260806 02:49:44.857661 31744 connection.cc:535] client connection to 127.0.0.1:36605 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36605): Connection reset by peer (error 104)
2026-08-06T02:49:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1625 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:49:44 dist-test-slave-gmjp krb5kdc[31919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:49:44 dist-test-slave-gmjp krb5kdc[31919](info): set up 2 sockets
Aug 06 02:49:44 dist-test-slave-gmjp krb5kdc[31919](info): commencing operation
krb5kdc: starting...
W20260806 02:49:46.074716 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.203s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:49:46 dist-test-slave-gmjp krb5kdc[31919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:49:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:49:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:49:46.205211 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41211
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:49:46.321141 31938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:46.321537 31938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:46.321592 31938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:46.325625 31938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:49:46.325708 31938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:46.325732 31938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:46.325752 31938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:49:46.325771 31938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:49:46.331110 31938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41211
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:46.332403 31938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:46.333467 31938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:46.342197 31949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:46.342267 31944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:46.342278 31945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:46.342594 31938 server_base.cc:1034] running on GCE node
I20260806 02:49:46.343154 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:46.344383 31938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:46.345582 31938 hybrid_clock.cc:648] HybridClock initialized: now 1785984586345568 us; error 25 us; skew 500 ppm
Aug 06 02:49:46 dist-test-slave-gmjp krb5kdc[31919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:46.348501 31938 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:49:46.349457 31938 webserver.cc:460] Webserver started at http://127.0.0.1:39993/ using document root <none> and password file <none>
I20260806 02:49:46.350068 31938 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:46.350170 31938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:46.350423 31938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:46.352659 31938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "b84855680f614b09b67b5aa84e642ff2"
format_stamp: "Formatted at 2026-08-06 02:49:46 on dist-test-slave-gmjp"
server_key: "4469df72759a438223d1728d106d6bea"
server_key_iv: "48e3b15d1dced9870b8889fd3a832362"
server_key_version: "encryptionkey@0"
I20260806 02:49:46.353282 31938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "b84855680f614b09b67b5aa84e642ff2"
format_stamp: "Formatted at 2026-08-06 02:49:46 on dist-test-slave-gmjp"
server_key: "4469df72759a438223d1728d106d6bea"
server_key_iv: "48e3b15d1dced9870b8889fd3a832362"
server_key_version: "encryptionkey@0"
I20260806 02:49:46.357141 31938 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:49:46.359709 31959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:46.361038 31938 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:49:46.361182 31938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "b84855680f614b09b67b5aa84e642ff2"
format_stamp: "Formatted at 2026-08-06 02:49:46 on dist-test-slave-gmjp"
server_key: "4469df72759a438223d1728d106d6bea"
server_key_iv: "48e3b15d1dced9870b8889fd3a832362"
server_key_version: "encryptionkey@0"
I20260806 02:49:46.361297 31938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:46.375917 31938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:46.379225 31938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:46.379437 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:46.386795 31938 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41211
I20260806 02:49:46.386950 32036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41211 every 8 connection(s)
I20260806 02:49:46.388118 31938 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:49:46.392052 32038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:46.392928 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 31938
I20260806 02:49:46.393080 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:49:46.393448 1758 external_mini_cluster.cc:1428] Setting key 6e43f5585fb069a809fb58a73a4741c0
I20260806 02:49:46.398682 32038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b84855680f614b09b67b5aa84e642ff2: Bootstrap starting.
I20260806 02:49:46.401018 32038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b84855680f614b09b67b5aa84e642ff2: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:46.401763 32038 log.cc:824] T 00000000000000000000000000000000 P b84855680f614b09b67b5aa84e642ff2: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:46.403002 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:46.394994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60069 (local address 127.0.0.1:41211)
0806 02:49:46.395479 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:46.395496 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:46.395534 (+ 38us) server_negotiation.cc:407] Connection header received
0806 02:49:46.396324 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:46.396344 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:46.396701 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:46.397047 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:46.397458 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:46.398424 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:46.399429 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:46.399898 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:46.400100 (+ 202us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:46.401106 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:46.401115 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:46.401133 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:49:46.401544 (+ 411us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:46.401600 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:49:46.401623 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:46.401774 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:46.401946 (+ 172us) server_negotiation.cc:299] Negotiation successful
0806 02:49:46.402093 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":144,"threads_started":1}
I20260806 02:49:46.403856 32038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b84855680f614b09b67b5aa84e642ff2: No bootstrap required, opened a new log
I20260806 02:49:46.405973 1758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:49:46.407799 32038 raft_consensus.cc:348] T 00000000000000000000000000000000 P b84855680f614b09b67b5aa84e642ff2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84855680f614b09b67b5aa84e642ff2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41211 } }
I20260806 02:49:46.408109 32038 raft_consensus.cc:374] T 00000000000000000000000000000000 P b84855680f614b09b67b5aa84e642ff2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:46.408195 32038 raft_consensus.cc:729] T 00000000000000000000000000000000 P b84855680f614b09b67b5aa84e642ff2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b84855680f614b09b67b5aa84e642ff2, State: Initialized, Role: FOLLOWER
I20260806 02:49:46.408715 32038 consensus_queue.cc:260] T 00000000000000000000000000000000 P b84855680f614b09b67b5aa84e642ff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84855680f614b09b67b5aa84e642ff2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41211 } }
I20260806 02:49:46.408895 32038 raft_consensus.cc:388] T 00000000000000000000000000000000 P b84855680f614b09b67b5aa84e642ff2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:49:46.408985 32038 raft_consensus.cc:482] T 00000000000000000000000000000000 P b84855680f614b09b67b5aa84e642ff2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:49:46.409107 32038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b84855680f614b09b67b5aa84e642ff2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:46.410218 32038 raft_consensus.cc:504] T 00000000000000000000000000000000 P b84855680f614b09b67b5aa84e642ff2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84855680f614b09b67b5aa84e642ff2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41211 } }
I20260806 02:49:46.410583 32038 leader_election.cc:302] T 00000000000000000000000000000000 P b84855680f614b09b67b5aa84e642ff2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b84855680f614b09b67b5aa84e642ff2; no voters:
I20260806 02:49:46.410972 32038 leader_election.cc:288] T 00000000000000000000000000000000 P b84855680f614b09b67b5aa84e642ff2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:49:46.411044 32054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b84855680f614b09b67b5aa84e642ff2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:49:46.411262 32054 raft_consensus.cc:686] T 00000000000000000000000000000000 P b84855680f614b09b67b5aa84e642ff2 [term 1 LEADER]: Becoming Leader. State: Replica: b84855680f614b09b67b5aa84e642ff2, State: Running, Role: LEADER
I20260806 02:49:46.411566 32054 consensus_queue.cc:237] T 00000000000000000000000000000000 P b84855680f614b09b67b5aa84e642ff2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84855680f614b09b67b5aa84e642ff2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41211 } }
I20260806 02:49:46.412361 32038 sys_catalog.cc:564] T 00000000000000000000000000000000 P b84855680f614b09b67b5aa84e642ff2 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:49:46.412784 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:46.408057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44755 (local address 127.0.0.1:41211)
0806 02:49:46.408208 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:46.408215 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:46.408447 (+ 232us) server_negotiation.cc:407] Connection header received
0806 02:49:46.408554 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:46.408561 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:46.408601 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:46.408668 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:46.408977 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:46.409581 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:46.410724 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:46.411050 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:46.411164 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:46.412251 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:46.412254 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:46.412257 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:49:46.412277 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:46.412302 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:49:46.412312 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:46.412413 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:46.412595 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 02:49:46.412632 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:49:46.414354 32057 sys_catalog.cc:455] T 00000000000000000000000000000000 P b84855680f614b09b67b5aa84e642ff2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b84855680f614b09b67b5aa84e642ff2. Latest consensus state: current_term: 1 leader_uuid: "b84855680f614b09b67b5aa84e642ff2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84855680f614b09b67b5aa84e642ff2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41211 } } }
I20260806 02:49:46.414496 32057 sys_catalog.cc:458] T 00000000000000000000000000000000 P b84855680f614b09b67b5aa84e642ff2 [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:46.415035 32058 sys_catalog.cc:455] T 00000000000000000000000000000000 P b84855680f614b09b67b5aa84e642ff2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b84855680f614b09b67b5aa84e642ff2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84855680f614b09b67b5aa84e642ff2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41211 } } }
I20260806 02:49:46.415167 32058 sys_catalog.cc:458] T 00000000000000000000000000000000 P b84855680f614b09b67b5aa84e642ff2 [sys.catalog]: This master's current role is: LEADER
W20260806 02:49:46.417470 32080 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b84855680f614b09b67b5aa84e642ff2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:49:46.417560 32080 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:49:46.417640 32079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:49:46.418305 32079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:49:46.423342 32079 catalog_manager.cc:1347] Generated new cluster ID: 6fa58d3cce924669bf32682c331f9b5e
I20260806 02:49:46.423408 32079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:49:46.434506 32079 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:49:46.435688 32079 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:49:46.443320 32079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b84855680f614b09b67b5aa84e642ff2: Generated new TSK 0
I20260806 02:49:46.444206 32079 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:49:46.456220 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 31938
W20260806 02:49:46.470934 31914 connection.cc:535] client connection to 127.0.0.1:41211 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41211): Connection reset by peer (error 104)
2026-08-06T02:49:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1613 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:49:46 dist-test-slave-gmjp krb5kdc[32087](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:49:46 dist-test-slave-gmjp krb5kdc[32087](info): set up 2 sockets
Aug 06 02:49:46 dist-test-slave-gmjp krb5kdc[32087](info): commencing operation
krb5kdc: starting...
W20260806 02:49:47.735476 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.250s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:49:47 dist-test-slave-gmjp krb5kdc[32087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:49:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:49:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:49:47.863772 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39695
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:49:47.978967 32102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:47.979279 32102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:47.979355 32102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:47.983367 32102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:49:47.983462 32102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:47.983502 32102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:47.983559 32102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:49:47.983600 32102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:49:47.988837 32102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39695
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:47.989887 32102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:47.990896 32102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:47.999346 32114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:47.999389 32112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:47.999675 32102 server_base.cc:1034] running on GCE node
W20260806 02:49:47.999346 32111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:48.000295 32102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:48.001367 32102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:48.002543 32102 hybrid_clock.cc:648] HybridClock initialized: now 1785984588002533 us; error 39 us; skew 500 ppm
Aug 06 02:49:48 dist-test-slave-gmjp krb5kdc[32087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:48.012924 32102 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:49:48.014004 32102 webserver.cc:460] Webserver started at http://127.0.0.1:41607/ using document root <none> and password file <none>
I20260806 02:49:48.014628 32102 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:48.014726 32102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:48.014986 32102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:48.017314 32102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "5a4a2df39dc84475a23ea236b8794140"
format_stamp: "Formatted at 2026-08-06 02:49:48 on dist-test-slave-gmjp"
server_key: "3dd0bc75426127d22a7a10db8a39f5c3"
server_key_iv: "fd7df987d0b285a0e3b305626f74f7f0"
server_key_version: "encryptionkey@0"
I20260806 02:49:48.017915 32102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "5a4a2df39dc84475a23ea236b8794140"
format_stamp: "Formatted at 2026-08-06 02:49:48 on dist-test-slave-gmjp"
server_key: "3dd0bc75426127d22a7a10db8a39f5c3"
server_key_iv: "fd7df987d0b285a0e3b305626f74f7f0"
server_key_version: "encryptionkey@0"
I20260806 02:49:48.021838 32102 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 02:49:48.024574 32123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:48.025817 32102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:49:48.025982 32102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "5a4a2df39dc84475a23ea236b8794140"
format_stamp: "Formatted at 2026-08-06 02:49:48 on dist-test-slave-gmjp"
server_key: "3dd0bc75426127d22a7a10db8a39f5c3"
server_key_iv: "fd7df987d0b285a0e3b305626f74f7f0"
server_key_version: "encryptionkey@0"
I20260806 02:49:48.026109 32102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:48.031071 32102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:48.034272 32102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:48.034488 32102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:48.041565 32102 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39695
I20260806 02:49:48.041592 32199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39695 every 8 connection(s)
I20260806 02:49:48.042474 32102 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:49:48.045252 32200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:48.050684 32200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a4a2df39dc84475a23ea236b8794140: Bootstrap starting.
I20260806 02:49:48.051054 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 32102
I20260806 02:49:48.051208 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:49:48.051478 1758 external_mini_cluster.cc:1428] Setting key 17fa965f684b0df800503af1a013dfe9
I20260806 02:49:48.053351 32200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a4a2df39dc84475a23ea236b8794140: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:48.054072 32200 log.cc:824] T 00000000000000000000000000000000 P 5a4a2df39dc84475a23ea236b8794140: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:48.056283 32200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a4a2df39dc84475a23ea236b8794140: No bootstrap required, opened a new log
I20260806 02:49:48.056784 32205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:48.052797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53543 (local address 127.0.0.1:39695)
0806 02:49:48.053246 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:48.053260 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:48.053294 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:49:48.053962 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:48.053979 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:48.054299 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:48.054649 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:48.055426 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:48.055430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:48.055442 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:49:48.055451 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:48.055498 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:49:48.055516 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:48.055600 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:48.055760 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 02:49:48.055945 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":133,"threads_started":1}
I20260806 02:49:48.059744 1758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:49:48.060575 32200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a4a2df39dc84475a23ea236b8794140 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a4a2df39dc84475a23ea236b8794140" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39695 } }
I20260806 02:49:48.060859 32200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a4a2df39dc84475a23ea236b8794140 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:48.060921 32200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a4a2df39dc84475a23ea236b8794140 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4a2df39dc84475a23ea236b8794140, State: Initialized, Role: FOLLOWER
I20260806 02:49:48.061409 32200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a4a2df39dc84475a23ea236b8794140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a4a2df39dc84475a23ea236b8794140" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39695 } }
I20260806 02:49:48.061574 32200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a4a2df39dc84475a23ea236b8794140 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:49:48.061643 32200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a4a2df39dc84475a23ea236b8794140 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:49:48.061741 32200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a4a2df39dc84475a23ea236b8794140 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:48.062968 32200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a4a2df39dc84475a23ea236b8794140 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a4a2df39dc84475a23ea236b8794140" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39695 } }
I20260806 02:49:48.063339 32200 leader_election.cc:302] T 00000000000000000000000000000000 P 5a4a2df39dc84475a23ea236b8794140 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a4a2df39dc84475a23ea236b8794140; no voters:
I20260806 02:49:48.063717 32200 leader_election.cc:288] T 00000000000000000000000000000000 P 5a4a2df39dc84475a23ea236b8794140 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:49:48.063812 32216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a4a2df39dc84475a23ea236b8794140 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:49:48.063934 32205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:48.062091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36025 (local address 127.0.0.1:39695)
0806 02:49:48.062220 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:48.062224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:48.062452 (+ 228us) server_negotiation.cc:407] Connection header received
0806 02:49:48.062559 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:48.062561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:48.062588 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:48.062662 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:48.063345 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:48.063347 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:48.063349 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:49:48.063359 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:48.063403 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:49:48.063413 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:48.063504 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:48.063593 (+ 89us) server_negotiation.cc:299] Negotiation successful
0806 02:49:48.063692 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:49:48.064303 32216 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a4a2df39dc84475a23ea236b8794140 [term 1 LEADER]: Becoming Leader. State: Replica: 5a4a2df39dc84475a23ea236b8794140, State: Running, Role: LEADER
I20260806 02:49:48.064607 32216 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a4a2df39dc84475a23ea236b8794140 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a4a2df39dc84475a23ea236b8794140" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39695 } }
I20260806 02:49:48.065126 32200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a4a2df39dc84475a23ea236b8794140 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:49:48.065918 32216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a4a2df39dc84475a23ea236b8794140 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a4a2df39dc84475a23ea236b8794140. Latest consensus state: current_term: 1 leader_uuid: "5a4a2df39dc84475a23ea236b8794140" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a4a2df39dc84475a23ea236b8794140" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39695 } } }
I20260806 02:49:48.066056 32216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a4a2df39dc84475a23ea236b8794140 [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:48.066468 32218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a4a2df39dc84475a23ea236b8794140 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a4a2df39dc84475a23ea236b8794140" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a4a2df39dc84475a23ea236b8794140" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39695 } } }
I20260806 02:49:48.066627 32218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a4a2df39dc84475a23ea236b8794140 [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:48.067011 32237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:49:48.069744 32237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:49:48.075330 32237 catalog_manager.cc:1347] Generated new cluster ID: 4b41a74c0ace435bb5d89e96bbebdd4b
I20260806 02:49:48.075402 32237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:49:48.086910 32237 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:49:48.087227 32237 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:49:48.094749 32237 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a4a2df39dc84475a23ea236b8794140: Generated new TSK 0
I20260806 02:49:48.095427 32237 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:49:48.103273 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 32102
2026-08-06T02:49:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1649 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:49:48 dist-test-slave-gmjp krb5kdc[32253](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:49:48 dist-test-slave-gmjp krb5kdc[32253](info): set up 2 sockets
Aug 06 02:49:48 dist-test-slave-gmjp krb5kdc[32253](info): commencing operation
krb5kdc: starting...
W20260806 02:49:49.339151 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.204s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:49:49 dist-test-slave-gmjp krb5kdc[32253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:49:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:49:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:49:49.472900 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41175
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:49:49.585048 32269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:49.585345 32269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:49.585412 32269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:49.589352 32269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:49:49.589440 32269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:49.589476 32269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:49.589531 32269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:49:49.589570 32269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:49:49.594686 32269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:49.595777 32269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:49.596849 32269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:49.605453 32277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:49.605443 32282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:49.605504 32269 server_base.cc:1034] running on GCE node
W20260806 02:49:49.605458 32280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:49.606338 32269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:49.607539 32269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:49.608729 32269 hybrid_clock.cc:648] HybridClock initialized: now 1785984589608701 us; error 46 us; skew 500 ppm
Aug 06 02:49:49 dist-test-slave-gmjp krb5kdc[32253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:49.611498 32269 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:49:49.612520 32269 webserver.cc:460] Webserver started at http://127.0.0.1:41229/ using document root <none> and password file <none>
I20260806 02:49:49.613090 32269 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:49.613179 32269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:49.613411 32269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:49.615473 32269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "399c8454bd4f42a0a0eb7ac24d453559"
format_stamp: "Formatted at 2026-08-06 02:49:49 on dist-test-slave-gmjp"
server_key: "21e15a02d42bec690cad76e3b1955113"
server_key_iv: "26a35e7e2b505f3f2906240e3bc4b660"
server_key_version: "encryptionkey@0"
I20260806 02:49:49.616051 32269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "399c8454bd4f42a0a0eb7ac24d453559"
format_stamp: "Formatted at 2026-08-06 02:49:49 on dist-test-slave-gmjp"
server_key: "21e15a02d42bec690cad76e3b1955113"
server_key_iv: "26a35e7e2b505f3f2906240e3bc4b660"
server_key_version: "encryptionkey@0"
I20260806 02:49:49.619789 32269 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:49:49.622268 32289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:49.623485 32269 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:49:49.623623 32269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "399c8454bd4f42a0a0eb7ac24d453559"
format_stamp: "Formatted at 2026-08-06 02:49:49 on dist-test-slave-gmjp"
server_key: "21e15a02d42bec690cad76e3b1955113"
server_key_iv: "26a35e7e2b505f3f2906240e3bc4b660"
server_key_version: "encryptionkey@0"
I20260806 02:49:49.623736 32269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:49.652319 32269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:49.655704 32269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:49.655923 32269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:49.663146 32269 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41175
I20260806 02:49:49.663174 32358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41175 every 8 connection(s)
I20260806 02:49:49.664093 32269 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:49:49.666819 32359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:49.670357 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 32269
I20260806 02:49:49.670497 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:49:49.670779 1758 external_mini_cluster.cc:1428] Setting key 0bcb7028fe01c64326875cc99bbf7b39
I20260806 02:49:49.672839 32359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 399c8454bd4f42a0a0eb7ac24d453559: Bootstrap starting.
I20260806 02:49:49.675017 32359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 399c8454bd4f42a0a0eb7ac24d453559: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:49.675953 32359 log.cc:824] T 00000000000000000000000000000000 P 399c8454bd4f42a0a0eb7ac24d453559: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:49.678035 32359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 399c8454bd4f42a0a0eb7ac24d453559: No bootstrap required, opened a new log
Aug 06 02:49:49 dist-test-slave-gmjp krb5kdc[32253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 02:49:49.682073 32359 raft_consensus.cc:348] T 00000000000000000000000000000000 P 399c8454bd4f42a0a0eb7ac24d453559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "399c8454bd4f42a0a0eb7ac24d453559" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41175 } }
I20260806 02:49:49.682327 32359 raft_consensus.cc:374] T 00000000000000000000000000000000 P 399c8454bd4f42a0a0eb7ac24d453559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:49.682410 32359 raft_consensus.cc:729] T 00000000000000000000000000000000 P 399c8454bd4f42a0a0eb7ac24d453559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 399c8454bd4f42a0a0eb7ac24d453559, State: Initialized, Role: FOLLOWER
I20260806 02:49:49.682844 32359 consensus_queue.cc:260] T 00000000000000000000000000000000 P 399c8454bd4f42a0a0eb7ac24d453559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "399c8454bd4f42a0a0eb7ac24d453559" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41175 } }
I20260806 02:49:49.682992 32359 raft_consensus.cc:388] T 00000000000000000000000000000000 P 399c8454bd4f42a0a0eb7ac24d453559 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:49:49.683073 32359 raft_consensus.cc:482] T 00000000000000000000000000000000 P 399c8454bd4f42a0a0eb7ac24d453559 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:49:49.683185 32359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 399c8454bd4f42a0a0eb7ac24d453559 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:49.684397 32359 raft_consensus.cc:504] T 00000000000000000000000000000000 P 399c8454bd4f42a0a0eb7ac24d453559 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "399c8454bd4f42a0a0eb7ac24d453559" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41175 } }
I20260806 02:49:49.684743 32359 leader_election.cc:302] T 00000000000000000000000000000000 P 399c8454bd4f42a0a0eb7ac24d453559 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 399c8454bd4f42a0a0eb7ac24d453559; no voters:
I20260806 02:49:49.685110 32359 leader_election.cc:288] T 00000000000000000000000000000000 P 399c8454bd4f42a0a0eb7ac24d453559 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:49:49.685225 32365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 399c8454bd4f42a0a0eb7ac24d453559 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:49:49.685515 32365 raft_consensus.cc:686] T 00000000000000000000000000000000 P 399c8454bd4f42a0a0eb7ac24d453559 [term 1 LEADER]: Becoming Leader. State: Replica: 399c8454bd4f42a0a0eb7ac24d453559, State: Running, Role: LEADER
I20260806 02:49:49.685902 32365 consensus_queue.cc:237] T 00000000000000000000000000000000 P 399c8454bd4f42a0a0eb7ac24d453559 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "399c8454bd4f42a0a0eb7ac24d453559" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41175 } }
I20260806 02:49:49.686470 32359 sys_catalog.cc:564] T 00000000000000000000000000000000 P 399c8454bd4f42a0a0eb7ac24d453559 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:49:49.686640 32363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:49.672298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56435 (local address 127.0.0.1:41175)
0806 02:49:49.672775 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:49.672792 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:49.672830 (+ 38us) server_negotiation.cc:407] Connection header received
0806 02:49:49.673524 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:49.673540 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:49.673889 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:49.674227 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:49.675126 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:49.676584 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:49.677478 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:49.678170 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:49.678329 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:49.681247 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:49.681252 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:49.681267 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:49.681294 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:49.683288 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:49.683795 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:49.683799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:49.683804 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:49.683913 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:49.684257 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:49.684260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:49.684262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:49.684724 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:49.684902 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:49.685296 (+ 394us) server_negotiation.cc:299] Negotiation successful
0806 02:49:49.685654 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":134,"threads_started":1}
I20260806 02:49:49.687116 32365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 399c8454bd4f42a0a0eb7ac24d453559 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 399c8454bd4f42a0a0eb7ac24d453559. Latest consensus state: current_term: 1 leader_uuid: "399c8454bd4f42a0a0eb7ac24d453559" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "399c8454bd4f42a0a0eb7ac24d453559" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41175 } } }
I20260806 02:49:49.687249 32365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 399c8454bd4f42a0a0eb7ac24d453559 [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:49.687667 32366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 399c8454bd4f42a0a0eb7ac24d453559 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "399c8454bd4f42a0a0eb7ac24d453559" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "399c8454bd4f42a0a0eb7ac24d453559" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41175 } } }
I20260806 02:49:49.687830 32366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 399c8454bd4f42a0a0eb7ac24d453559 [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:49.688628 32373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:49:49.690080 1758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:49:49.691934 32373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:49:49.698400 32373 catalog_manager.cc:1347] Generated new cluster ID: 2ca7c4749c874162a2584606a2e8e5b8
I20260806 02:49:49.698480 32373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:49:49.701078 32363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:49.692197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53043 (local address 127.0.0.1:41175)
0806 02:49:49.692303 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:49.692308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:49.692690 (+ 382us) server_negotiation.cc:407] Connection header received
0806 02:49:49.692848 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:49.692851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:49.692898 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:49.692974 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:49.693625 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:49.694357 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:49.695524 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:49.695842 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:49.695955 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:49.697583 (+ 1628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:49.697587 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:49.697591 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:49.697616 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:49.699304 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:49.699809 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:49.699813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:49.699815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:49.699916 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:49.700208 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:49.700212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:49.700214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:49.700426 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:49.700561 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:49.700789 (+ 228us) server_negotiation.cc:299] Negotiation successful
0806 02:49:49.700920 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 02:49:49.703975 32373 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:49:49.705120 32373 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:49:49.718758 32373 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 399c8454bd4f42a0a0eb7ac24d453559: Generated new TSK 0
I20260806 02:49:49.719655 32373 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:49:49.728605 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 32269
2026-08-06T02:49:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1623 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:49:49 dist-test-slave-gmjp krb5kdc[32404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:49:49 dist-test-slave-gmjp krb5kdc[32404](info): set up 2 sockets
Aug 06 02:49:49 dist-test-slave-gmjp krb5kdc[32404](info): commencing operation
krb5kdc: starting...
W20260806 02:49:50.957602 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:49:51 dist-test-slave-gmjp krb5kdc[32404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:49:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:49:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:49:51.087335 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46877
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:49:51.206243 32427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:51.206511 32427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:51.206553 32427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:51.210528 32427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:49:51.210587 32427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:51.210618 32427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:51.210633 32427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:49:51.210654 32427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:49:51.215818 32427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:51.216857 32427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:51.217808 32427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:51.226266 32435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:51.226318 32439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:51.226294 32434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:51.226537 32427 server_base.cc:1034] running on GCE node
I20260806 02:49:51.227185 32427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:51.228487 32427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:51.229691 32427 hybrid_clock.cc:648] HybridClock initialized: now 1785984591229656 us; error 47 us; skew 500 ppm
Aug 06 02:49:51 dist-test-slave-gmjp krb5kdc[32404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:51.232524 32427 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:49:51.233486 32427 webserver.cc:460] Webserver started at http://127.0.0.1:35135/ using document root <none> and password file <none>
I20260806 02:49:51.234071 32427 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:51.234164 32427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:51.234393 32427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:51.236515 32427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "5bd439886d6a47e79c8127ba8d898d29"
format_stamp: "Formatted at 2026-08-06 02:49:51 on dist-test-slave-gmjp"
server_key: "481dbfea7b87fbea4c3759f262035df5"
server_key_iv: "85dc534c5b865396a2c3b8cfd4a67485"
server_key_version: "encryptionkey@0"
I20260806 02:49:51.237090 32427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "5bd439886d6a47e79c8127ba8d898d29"
format_stamp: "Formatted at 2026-08-06 02:49:51 on dist-test-slave-gmjp"
server_key: "481dbfea7b87fbea4c3759f262035df5"
server_key_iv: "85dc534c5b865396a2c3b8cfd4a67485"
server_key_version: "encryptionkey@0"
I20260806 02:49:51.241240 32427 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:49:51.243898 32447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:51.245208 32427 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 02:49:51.245368 32427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "5bd439886d6a47e79c8127ba8d898d29"
format_stamp: "Formatted at 2026-08-06 02:49:51 on dist-test-slave-gmjp"
server_key: "481dbfea7b87fbea4c3759f262035df5"
server_key_iv: "85dc534c5b865396a2c3b8cfd4a67485"
server_key_version: "encryptionkey@0"
I20260806 02:49:51.245491 32427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:51.258914 32427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:51.262389 32427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:51.262617 32427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:51.270321 32427 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46877
I20260806 02:49:51.270349 32516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46877 every 8 connection(s)
I20260806 02:49:51.271315 32427 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:49:51.273957 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 32427
I20260806 02:49:51.274106 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:49:51.274155 32518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:51.274407 1758 external_mini_cluster.cc:1428] Setting key 623795c051add1c0661d73d8482977df
I20260806 02:49:51.281579 32518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bd439886d6a47e79c8127ba8d898d29: Bootstrap starting.
I20260806 02:49:51.284296 32518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bd439886d6a47e79c8127ba8d898d29: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:51.284662 32524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:51.275868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40589 (local address 127.0.0.1:46877)
0806 02:49:51.276348 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:51.276365 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:51.276419 (+ 54us) server_negotiation.cc:407] Connection header received
0806 02:49:51.277101 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:51.277123 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:51.277583 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:51.277958 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:51.278374 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:51.279849 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:51.280899 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:51.281576 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:51.281742 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:51.282595 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:51.282604 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:51.282622 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:49:51.283138 (+ 516us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:51.283204 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:49:51.283232 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:51.283324 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:51.283469 (+ 145us) server_negotiation.cc:299] Negotiation successful
0806 02:49:51.283677 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":127,"threads_started":1}
I20260806 02:49:51.285182 32518 log.cc:824] T 00000000000000000000000000000000 P 5bd439886d6a47e79c8127ba8d898d29: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:51.287344 32518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bd439886d6a47e79c8127ba8d898d29: No bootstrap required, opened a new log
I20260806 02:49:51.287549 1758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:49:51.291499 32518 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bd439886d6a47e79c8127ba8d898d29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd439886d6a47e79c8127ba8d898d29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46877 } }
I20260806 02:49:51.291724 32518 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bd439886d6a47e79c8127ba8d898d29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:51.291791 32518 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bd439886d6a47e79c8127ba8d898d29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bd439886d6a47e79c8127ba8d898d29, State: Initialized, Role: FOLLOWER
I20260806 02:49:51.292318 32518 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bd439886d6a47e79c8127ba8d898d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd439886d6a47e79c8127ba8d898d29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46877 } }
I20260806 02:49:51.292456 32518 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bd439886d6a47e79c8127ba8d898d29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:49:51.292546 32518 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bd439886d6a47e79c8127ba8d898d29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:49:51.292662 32518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bd439886d6a47e79c8127ba8d898d29 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:51.293910 32518 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bd439886d6a47e79c8127ba8d898d29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd439886d6a47e79c8127ba8d898d29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46877 } }
I20260806 02:49:51.294363 32518 leader_election.cc:302] T 00000000000000000000000000000000 P 5bd439886d6a47e79c8127ba8d898d29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bd439886d6a47e79c8127ba8d898d29; no voters:
I20260806 02:49:51.294626 32524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:51.289957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57035 (local address 127.0.0.1:46877)
0806 02:49:51.290101 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:51.290107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:51.290326 (+ 219us) server_negotiation.cc:407] Connection header received
0806 02:49:51.290484 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:51.290488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:51.290538 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:51.290630 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:51.290925 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:51.291490 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:51.292666 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:51.293021 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:51.293134 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:51.294018 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:51.294025 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:51.294029 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:49:51.294044 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:51.294074 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:49:51.294087 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:51.294168 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:51.294369 (+ 201us) server_negotiation.cc:299] Negotiation successful
0806 02:49:51.294427 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:49:51.294775 32518 leader_election.cc:288] T 00000000000000000000000000000000 P 5bd439886d6a47e79c8127ba8d898d29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:49:51.295202 32538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bd439886d6a47e79c8127ba8d898d29 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:49:51.295480 32538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bd439886d6a47e79c8127ba8d898d29 [term 1 LEADER]: Becoming Leader. State: Replica: 5bd439886d6a47e79c8127ba8d898d29, State: Running, Role: LEADER
I20260806 02:49:51.295799 32538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bd439886d6a47e79c8127ba8d898d29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd439886d6a47e79c8127ba8d898d29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46877 } }
I20260806 02:49:51.296257 32518 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bd439886d6a47e79c8127ba8d898d29 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:49:51.297555 32537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bd439886d6a47e79c8127ba8d898d29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bd439886d6a47e79c8127ba8d898d29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd439886d6a47e79c8127ba8d898d29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46877 } } }
I20260806 02:49:51.297689 32537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bd439886d6a47e79c8127ba8d898d29 [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:51.298100 32549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:49:51.298408 32539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bd439886d6a47e79c8127ba8d898d29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bd439886d6a47e79c8127ba8d898d29. Latest consensus state: current_term: 1 leader_uuid: "5bd439886d6a47e79c8127ba8d898d29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd439886d6a47e79c8127ba8d898d29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46877 } } }
I20260806 02:49:51.299294 32539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bd439886d6a47e79c8127ba8d898d29 [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:51.301462 32549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:49:51.306715 32549 catalog_manager.cc:1347] Generated new cluster ID: 12008522775c4d61980dac18e1041503
I20260806 02:49:51.306782 32549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:49:51.334501 32549 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:49:51.335490 32549 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:49:51.341711 32549 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bd439886d6a47e79c8127ba8d898d29: Generated new TSK 0
I20260806 02:49:51.342414 32549 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:49:51.351920 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 32427
2026-08-06T02:49:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1616 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:49:51 dist-test-slave-gmjp krb5kdc[32564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:49:51 dist-test-slave-gmjp krb5kdc[32564](info): set up 2 sockets
Aug 06 02:49:51 dist-test-slave-gmjp krb5kdc[32564](info): commencing operation
krb5kdc: starting...
W20260806 02:49:52.587947 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.213s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:49:52 dist-test-slave-gmjp krb5kdc[32564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:49:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:49:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:49:52.718713 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34411
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:49:52.832549 32580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:52.832825 32580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:52.832867 32580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:52.836983 32580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:49:52.837040 32580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:52.837060 32580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:52.837077 32580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:49:52.837092 32580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:49:52.842525 32580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:52.843531 32580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:52.844591 32580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:52.852846 32591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:52.852872 32593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:52.852846 32588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:52.853044 32580 server_base.cc:1034] running on GCE node
I20260806 02:49:52.853647 32580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:52.854737 32580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:52.855932 32580 hybrid_clock.cc:648] HybridClock initialized: now 1785984592855907 us; error 40 us; skew 500 ppm
Aug 06 02:49:52 dist-test-slave-gmjp krb5kdc[32564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984592, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:52.858789 32580 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:49:52.859673 32580 webserver.cc:460] Webserver started at http://127.0.0.1:37117/ using document root <none> and password file <none>
I20260806 02:49:52.860239 32580 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:52.860304 32580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:52.860474 32580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:52.862468 32580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "2befc643cba84484820d43dd20e44f5f"
format_stamp: "Formatted at 2026-08-06 02:49:52 on dist-test-slave-gmjp"
server_key: "8f4ec1437ca2809c82f002d8a3fc4888"
server_key_iv: "40e8438fe7f075981bced1377d27fda0"
server_key_version: "encryptionkey@0"
I20260806 02:49:52.862994 32580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "2befc643cba84484820d43dd20e44f5f"
format_stamp: "Formatted at 2026-08-06 02:49:52 on dist-test-slave-gmjp"
server_key: "8f4ec1437ca2809c82f002d8a3fc4888"
server_key_iv: "40e8438fe7f075981bced1377d27fda0"
server_key_version: "encryptionkey@0"
I20260806 02:49:52.866710 32580 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 02:49:52.869076 32609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:52.870271 32580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:49:52.870368 32580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "2befc643cba84484820d43dd20e44f5f"
format_stamp: "Formatted at 2026-08-06 02:49:52 on dist-test-slave-gmjp"
server_key: "8f4ec1437ca2809c82f002d8a3fc4888"
server_key_iv: "40e8438fe7f075981bced1377d27fda0"
server_key_version: "encryptionkey@0"
I20260806 02:49:52.870448 32580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:52.875073 32580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:52.878333 32580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:52.878489 32580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:52.885668 32580 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34411
I20260806 02:49:52.885689 32678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34411 every 8 connection(s)
I20260806 02:49:52.886607 32580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:49:52.889333 32679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:52.894721 32679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2befc643cba84484820d43dd20e44f5f: Bootstrap starting.
I20260806 02:49:52.895751 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 32580
I20260806 02:49:52.895927 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:49:52.896260 1758 external_mini_cluster.cc:1428] Setting key a564eb695688aab6a8da28f289d662a2
I20260806 02:49:52.897652 32679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2befc643cba84484820d43dd20e44f5f: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:52.898356 32679 log.cc:824] T 00000000000000000000000000000000 P 2befc643cba84484820d43dd20e44f5f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:52.900429 32679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2befc643cba84484820d43dd20e44f5f: No bootstrap required, opened a new log
I20260806 02:49:52.901365 32682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:52.897364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41669 (local address 127.0.0.1:34411)
0806 02:49:52.897856 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:52.897877 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:52.897912 (+ 35us) server_negotiation.cc:407] Connection header received
0806 02:49:52.898616 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:52.898635 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:52.898997 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:52.899375 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:52.900201 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:52.900206 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:52.900218 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:49:52.900229 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:52.900280 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:49:52.900300 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:52.900419 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:52.900523 (+ 104us) server_negotiation.cc:299] Negotiation successful
0806 02:49:52.900678 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":142,"threads_started":1}
I20260806 02:49:52.903977 1758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:49:52.904249 32679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2befc643cba84484820d43dd20e44f5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2befc643cba84484820d43dd20e44f5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34411 } }
I20260806 02:49:52.904488 32679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2befc643cba84484820d43dd20e44f5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:52.904559 32679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2befc643cba84484820d43dd20e44f5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2befc643cba84484820d43dd20e44f5f, State: Initialized, Role: FOLLOWER
I20260806 02:49:52.904963 32679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2befc643cba84484820d43dd20e44f5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2befc643cba84484820d43dd20e44f5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34411 } }
I20260806 02:49:52.905105 32679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2befc643cba84484820d43dd20e44f5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:49:52.905184 32679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2befc643cba84484820d43dd20e44f5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:49:52.905279 32679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2befc643cba84484820d43dd20e44f5f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:52.906478 32679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2befc643cba84484820d43dd20e44f5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2befc643cba84484820d43dd20e44f5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34411 } }
I20260806 02:49:52.906813 32679 leader_election.cc:302] T 00000000000000000000000000000000 P 2befc643cba84484820d43dd20e44f5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2befc643cba84484820d43dd20e44f5f; no voters:
I20260806 02:49:52.907160 32679 leader_election.cc:288] T 00000000000000000000000000000000 P 2befc643cba84484820d43dd20e44f5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:49:52.907251 32693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2befc643cba84484820d43dd20e44f5f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:49:52.907483 32693 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2befc643cba84484820d43dd20e44f5f [term 1 LEADER]: Becoming Leader. State: Replica: 2befc643cba84484820d43dd20e44f5f, State: Running, Role: LEADER
I20260806 02:49:52.907812 32693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2befc643cba84484820d43dd20e44f5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2befc643cba84484820d43dd20e44f5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34411 } }
I20260806 02:49:52.907974 32682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:52.906182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46863 (local address 127.0.0.1:34411)
0806 02:49:52.906288 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:52.906294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:52.906487 (+ 193us) server_negotiation.cc:407] Connection header received
0806 02:49:52.906614 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:52.906617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:52.906647 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:52.906733 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:52.907521 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:52.907525 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:52.907528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:49:52.907541 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:52.907570 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:49:52.907582 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:52.907641 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:52.907810 (+ 169us) server_negotiation.cc:299] Negotiation successful
0806 02:49:52.907852 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 02:49:52.908572 32679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2befc643cba84484820d43dd20e44f5f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:49:52.909669 32694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2befc643cba84484820d43dd20e44f5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2befc643cba84484820d43dd20e44f5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2befc643cba84484820d43dd20e44f5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34411 } } }
I20260806 02:49:52.909816 32694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2befc643cba84484820d43dd20e44f5f [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:52.910153 32695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2befc643cba84484820d43dd20e44f5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2befc643cba84484820d43dd20e44f5f. Latest consensus state: current_term: 1 leader_uuid: "2befc643cba84484820d43dd20e44f5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2befc643cba84484820d43dd20e44f5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34411 } } }
I20260806 02:49:52.910247 32695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2befc643cba84484820d43dd20e44f5f [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:52.910233 32705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:49:52.913034 32705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:49:52.918290 32705 catalog_manager.cc:1347] Generated new cluster ID: 1e19c9d25725400886157321b7f1b99d
I20260806 02:49:52.918365 32705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:49:52.932490 32705 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:49:52.932829 32705 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:49:52.943368 32705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2befc643cba84484820d43dd20e44f5f: Generated new TSK 0
I20260806 02:49:52.944056 32705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:49:52.955687 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 32580
2026-08-06T02:49:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1615 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:49:53 dist-test-slave-gmjp krb5kdc[32719](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:49:53 dist-test-slave-gmjp krb5kdc[32719](info): set up 2 sockets
Aug 06 02:49:53 dist-test-slave-gmjp krb5kdc[32719](info): commencing operation
krb5kdc: starting...
W20260806 02:49:54.535295 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.545s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:49:54 dist-test-slave-gmjp krb5kdc[32719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:49:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:49:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:49:54.667101 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38385
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:49:54.781133 32734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:54.781438 32734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:54.781509 32734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:54.785535 32734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:49:54.785629 32734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:54.785668 32734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:54.785722 32734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:49:54.785763 32734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:49:54.791152 32734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:54.792414 32734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:54.793496 32734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:49:54.802356 32734 server_base.cc:1034] running on GCE node
W20260806 02:49:54.802289 32742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:54.802280 32740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:54.802280 32745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:54.803232 32734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:54.804425 32734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:54.805603 32734 hybrid_clock.cc:648] HybridClock initialized: now 1785984594805576 us; error 39 us; skew 500 ppm
Aug 06 02:49:54 dist-test-slave-gmjp krb5kdc[32719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:54.808475 32734 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:49:54.809412 32734 webserver.cc:460] Webserver started at http://127.0.0.1:46053/ using document root <none> and password file <none>
I20260806 02:49:54.809983 32734 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:54.810073 32734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:54.810305 32734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:54.812496 32734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "1ae4f47c99ad49c7a6e1c3ee9e5152b1"
format_stamp: "Formatted at 2026-08-06 02:49:54 on dist-test-slave-gmjp"
server_key: "98e205a1b2abdcf0cccf23c3fbec5672"
server_key_iv: "f1aa5c4673a65e803c87225e48d5a17f"
server_key_version: "encryptionkey@0"
I20260806 02:49:54.813114 32734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "1ae4f47c99ad49c7a6e1c3ee9e5152b1"
format_stamp: "Formatted at 2026-08-06 02:49:54 on dist-test-slave-gmjp"
server_key: "98e205a1b2abdcf0cccf23c3fbec5672"
server_key_iv: "f1aa5c4673a65e803c87225e48d5a17f"
server_key_version: "encryptionkey@0"
I20260806 02:49:54.816962 32734 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260806 02:49:54.819449 32755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:54.820662 32734 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:49:54.820796 32734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "1ae4f47c99ad49c7a6e1c3ee9e5152b1"
format_stamp: "Formatted at 2026-08-06 02:49:54 on dist-test-slave-gmjp"
server_key: "98e205a1b2abdcf0cccf23c3fbec5672"
server_key_iv: "f1aa5c4673a65e803c87225e48d5a17f"
server_key_version: "encryptionkey@0"
I20260806 02:49:54.820910 32734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:54.831848 32734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:54.835327 32734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:54.835573 32734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:54.843071 32734 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38385
I20260806 02:49:54.843101 364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38385 every 8 connection(s)
I20260806 02:49:54.844106 32734 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:49:54.846933 366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:54.852516 366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ae4f47c99ad49c7a6e1c3ee9e5152b1: Bootstrap starting.
I20260806 02:49:54.853675 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 32734
I20260806 02:49:54.853875 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:49:54.854281 1758 external_mini_cluster.cc:1428] Setting key b2c82f8b9881f6dae6e509e9d1c67c58
I20260806 02:49:54.855320 366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ae4f47c99ad49c7a6e1c3ee9e5152b1: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:54.856251 366 log.cc:824] T 00000000000000000000000000000000 P 1ae4f47c99ad49c7a6e1c3ee9e5152b1: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:54.858526 366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ae4f47c99ad49c7a6e1c3ee9e5152b1: No bootstrap required, opened a new log
I20260806 02:49:54.862702 366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ae4f47c99ad49c7a6e1c3ee9e5152b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae4f47c99ad49c7a6e1c3ee9e5152b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38385 } }
I20260806 02:49:54.862923 366 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ae4f47c99ad49c7a6e1c3ee9e5152b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:54.862980 366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ae4f47c99ad49c7a6e1c3ee9e5152b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae4f47c99ad49c7a6e1c3ee9e5152b1, State: Initialized, Role: FOLLOWER
I20260806 02:49:54.863456 366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ae4f47c99ad49c7a6e1c3ee9e5152b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae4f47c99ad49c7a6e1c3ee9e5152b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38385 } }
I20260806 02:49:54.863621 366 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ae4f47c99ad49c7a6e1c3ee9e5152b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:49:54.863691 366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ae4f47c99ad49c7a6e1c3ee9e5152b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:49:54.863801 366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ae4f47c99ad49c7a6e1c3ee9e5152b1 [term 0 FOLLOWER]: Advancing to term 1
Aug 06 02:49:54 dist-test-slave-gmjp krb5kdc[32719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 02:49:54.864918 366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ae4f47c99ad49c7a6e1c3ee9e5152b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae4f47c99ad49c7a6e1c3ee9e5152b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38385 } }
I20260806 02:49:54.865335 366 leader_election.cc:302] T 00000000000000000000000000000000 P 1ae4f47c99ad49c7a6e1c3ee9e5152b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ae4f47c99ad49c7a6e1c3ee9e5152b1; no voters:
I20260806 02:49:54.865713 366 leader_election.cc:288] T 00000000000000000000000000000000 P 1ae4f47c99ad49c7a6e1c3ee9e5152b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:49:54.866010 377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ae4f47c99ad49c7a6e1c3ee9e5152b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:49:54.866263 377 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ae4f47c99ad49c7a6e1c3ee9e5152b1 [term 1 LEADER]: Becoming Leader. State: Replica: 1ae4f47c99ad49c7a6e1c3ee9e5152b1, State: Running, Role: LEADER
I20260806 02:49:54.866597 377 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ae4f47c99ad49c7a6e1c3ee9e5152b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae4f47c99ad49c7a6e1c3ee9e5152b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38385 } }
I20260806 02:49:54.867184 366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ae4f47c99ad49c7a6e1c3ee9e5152b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:49:54.867885 379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ae4f47c99ad49c7a6e1c3ee9e5152b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ae4f47c99ad49c7a6e1c3ee9e5152b1. Latest consensus state: current_term: 1 leader_uuid: "1ae4f47c99ad49c7a6e1c3ee9e5152b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae4f47c99ad49c7a6e1c3ee9e5152b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38385 } } }
I20260806 02:49:54.868079 379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ae4f47c99ad49c7a6e1c3ee9e5152b1 [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:54.868412 378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ae4f47c99ad49c7a6e1c3ee9e5152b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ae4f47c99ad49c7a6e1c3ee9e5152b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae4f47c99ad49c7a6e1c3ee9e5152b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38385 } } }
I20260806 02:49:54.868525 378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ae4f47c99ad49c7a6e1c3ee9e5152b1 [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:54.868633 388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:49:54.871219 374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:54.855434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:47635 (local address 127.0.0.1:38385)
0806 02:49:54.855940 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:54.855959 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:54.856005 (+ 46us) server_negotiation.cc:407] Connection header received
0806 02:49:54.856888 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:54.856908 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:54.857337 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:54.857710 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:54.858461 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:54.859912 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:54.860811 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:54.861618 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:54.861786 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:54.864682 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:54.864688 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:54.864707 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:54.864744 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:54.866859 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:54.867474 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:54.867480 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:54.867485 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:54.867631 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:54.868070 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:54.868074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:54.868079 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:54.868670 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:54.868910 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:54.869960 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0806 02:49:54.870208 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":169,"threads_started":1}
I20260806 02:49:54.871688 388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:49:54.874610 1758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:49:54.877651 388 catalog_manager.cc:1347] Generated new cluster ID: 5e77976313ea4b58afe05dbfcd1a0d7f
I20260806 02:49:54.877712 388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:49:54.886601 374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:54.876728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:32791 (local address 127.0.0.1:38385)
0806 02:49:54.876846 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:54.876851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:54.877768 (+ 917us) server_negotiation.cc:407] Connection header received
0806 02:49:54.877955 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:54.877959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:54.878010 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:54.878100 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:54.878953 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:54.879723 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:54.880916 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:54.881269 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:54.881360 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:54.883073 (+ 1713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:54.883076 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:54.883079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:54.883106 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:54.884782 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:54.885316 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:54.885319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:54.885321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:54.885399 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:54.885726 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:54.885729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:54.885731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:54.885923 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:54.886070 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:54.886349 (+ 279us) server_negotiation.cc:299] Negotiation successful
0806 02:49:54.886445 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:49:54.901156 388 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:49:54.902084 388 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:49:54.907452 388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ae4f47c99ad49c7a6e1c3ee9e5152b1: Generated new TSK 0
I20260806 02:49:54.908138 388 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:49:54.913771 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 32734
W20260806 02:49:54.935051 32717 connection.cc:535] client connection to 127.0.0.1:38385 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38385): Connection reset by peer (error 104)
2026-08-06T02:49:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1959 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:49:54 dist-test-slave-gmjp krb5kdc[412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:49:54 dist-test-slave-gmjp krb5kdc[412](info): set up 2 sockets
Aug 06 02:49:54 dist-test-slave-gmjp krb5kdc[412](info): commencing operation
krb5kdc: starting...
W20260806 02:49:56.149488 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.200s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:49:56 dist-test-slave-gmjp krb5kdc[412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:49:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:49:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:49:56.279356 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35777
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:49:56.395195 429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:56.395479 429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:56.395521 429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:56.399501 429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:49:56.399573 429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:56.399595 429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:56.399612 429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:49:56.399627 429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:49:56.404944 429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:56.406113 429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:56.407130 429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:56.416267 444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:56.416337 442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:56.416339 441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:56.416702 429 server_base.cc:1034] running on GCE node
I20260806 02:49:56.417287 429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:56.418495 429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:56.419659 429 hybrid_clock.cc:648] HybridClock initialized: now 1785984596419641 us; error 31 us; skew 500 ppm
Aug 06 02:49:56 dist-test-slave-gmjp krb5kdc[412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:56.422372 429 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:49:56.423417 429 webserver.cc:460] Webserver started at http://127.0.0.1:33187/ using document root <none> and password file <none>
I20260806 02:49:56.424113 429 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:56.424247 429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:56.424504 429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:56.427091 429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "a0f42ee592f54989a779c9998ac0b5bc"
format_stamp: "Formatted at 2026-08-06 02:49:56 on dist-test-slave-gmjp"
server_key: "126e6d0a187ddb1bd5d8428cec036f58"
server_key_iv: "1ef516b320aacd5df7f5bdb33118cb0e"
server_key_version: "encryptionkey@0"
I20260806 02:49:56.427691 429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "a0f42ee592f54989a779c9998ac0b5bc"
format_stamp: "Formatted at 2026-08-06 02:49:56 on dist-test-slave-gmjp"
server_key: "126e6d0a187ddb1bd5d8428cec036f58"
server_key_iv: "1ef516b320aacd5df7f5bdb33118cb0e"
server_key_version: "encryptionkey@0"
I20260806 02:49:56.431471 429 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 02:49:56.433930 454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:56.435113 429 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 02:49:56.435245 429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "a0f42ee592f54989a779c9998ac0b5bc"
format_stamp: "Formatted at 2026-08-06 02:49:56 on dist-test-slave-gmjp"
server_key: "126e6d0a187ddb1bd5d8428cec036f58"
server_key_iv: "1ef516b320aacd5df7f5bdb33118cb0e"
server_key_version: "encryptionkey@0"
I20260806 02:49:56.435370 429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:56.446449 429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:56.449357 429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:56.449566 429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:56.456858 429 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35777
I20260806 02:49:56.456944 529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35777 every 8 connection(s)
I20260806 02:49:56.457865 429 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:49:56.460695 530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:56.466423 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 429
I20260806 02:49:56.466585 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:49:56.466789 530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0f42ee592f54989a779c9998ac0b5bc: Bootstrap starting.
I20260806 02:49:56.466935 1758 external_mini_cluster.cc:1428] Setting key 384447203257f131fff268a6c6294572
I20260806 02:49:56.469672 530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0f42ee592f54989a779c9998ac0b5bc: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:56.470575 530 log.cc:824] T 00000000000000000000000000000000 P a0f42ee592f54989a779c9998ac0b5bc: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:56.472832 530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0f42ee592f54989a779c9998ac0b5bc: No bootstrap required, opened a new log
I20260806 02:49:56.477345 534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:56.468328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:51977 (local address 127.0.0.1:35777)
0806 02:49:56.468764 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:56.468787 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:56.468832 (+ 45us) server_negotiation.cc:407] Connection header received
0806 02:49:56.469722 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:56.469761 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:56.470215 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:56.470592 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:56.471052 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:56.472437 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:56.473395 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:56.474081 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:56.474257 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:56.475102 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:56.475111 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:56.475129 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:49:56.475684 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:56.475784 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:49:56.475822 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:56.476001 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:56.476178 (+ 177us) server_negotiation.cc:299] Negotiation successful
0806 02:49:56.476328 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":106,"threads_started":1}
I20260806 02:49:56.477428 530 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0f42ee592f54989a779c9998ac0b5bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0f42ee592f54989a779c9998ac0b5bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35777 } }
I20260806 02:49:56.477713 530 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0f42ee592f54989a779c9998ac0b5bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:56.477797 530 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0f42ee592f54989a779c9998ac0b5bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0f42ee592f54989a779c9998ac0b5bc, State: Initialized, Role: FOLLOWER
I20260806 02:49:56.478250 530 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0f42ee592f54989a779c9998ac0b5bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0f42ee592f54989a779c9998ac0b5bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35777 } }
I20260806 02:49:56.478410 530 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0f42ee592f54989a779c9998ac0b5bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:49:56.478518 530 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0f42ee592f54989a779c9998ac0b5bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:49:56.478622 530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0f42ee592f54989a779c9998ac0b5bc [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:56.479825 530 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0f42ee592f54989a779c9998ac0b5bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0f42ee592f54989a779c9998ac0b5bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35777 } }
I20260806 02:49:56.480264 530 leader_election.cc:302] T 00000000000000000000000000000000 P a0f42ee592f54989a779c9998ac0b5bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0f42ee592f54989a779c9998ac0b5bc; no voters:
I20260806 02:49:56.480628 530 leader_election.cc:288] T 00000000000000000000000000000000 P a0f42ee592f54989a779c9998ac0b5bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:49:56.480746 537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0f42ee592f54989a779c9998ac0b5bc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:49:56.480885 1758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:49:56.480993 537 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0f42ee592f54989a779c9998ac0b5bc [term 1 LEADER]: Becoming Leader. State: Replica: a0f42ee592f54989a779c9998ac0b5bc, State: Running, Role: LEADER
I20260806 02:49:56.481323 537 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0f42ee592f54989a779c9998ac0b5bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0f42ee592f54989a779c9998ac0b5bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35777 } }
I20260806 02:49:56.482034 530 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0f42ee592f54989a779c9998ac0b5bc [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:49:56.482473 537 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0f42ee592f54989a779c9998ac0b5bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0f42ee592f54989a779c9998ac0b5bc. Latest consensus state: current_term: 1 leader_uuid: "a0f42ee592f54989a779c9998ac0b5bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0f42ee592f54989a779c9998ac0b5bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35777 } } }
I20260806 02:49:56.482594 537 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0f42ee592f54989a779c9998ac0b5bc [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:56.482956 541 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0f42ee592f54989a779c9998ac0b5bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0f42ee592f54989a779c9998ac0b5bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0f42ee592f54989a779c9998ac0b5bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35777 } } }
I20260806 02:49:56.483057 541 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0f42ee592f54989a779c9998ac0b5bc [sys.catalog]: This master's current role is: LEADER
W20260806 02:49:56.487362 570 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a0f42ee592f54989a779c9998ac0b5bc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:49:56.487452 570 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:49:56.487524 550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:49:56.488477 550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:49:56.488801 534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:56.483305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:52349 (local address 127.0.0.1:35777)
0806 02:49:56.483420 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:56.483424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:56.483958 (+ 534us) server_negotiation.cc:407] Connection header received
0806 02:49:56.484250 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:56.484254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:56.484312 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:56.484401 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:56.484840 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:56.485580 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:56.486745 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:56.487100 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:56.487175 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:56.488292 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:56.488295 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:56.488298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:49:56.488312 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:56.488348 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:49:56.488359 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:56.488459 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:56.488611 (+ 152us) server_negotiation.cc:299] Negotiation successful
0806 02:49:56.488647 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 02:49:56.493667 550 catalog_manager.cc:1347] Generated new cluster ID: 09375bc5d2b14a8a9c00661d1afbcbb2
I20260806 02:49:56.493742 550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:49:56.511756 550 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:49:56.512753 550 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:49:56.517292 550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0f42ee592f54989a779c9998ac0b5bc: Generated new TSK 0
I20260806 02:49:56.517994 550 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:49:56.525233 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 429
W20260806 02:49:56.542214 409 connection.cc:535] client connection to 127.0.0.1:35777 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35777): Connection reset by peer (error 104)
2026-08-06T02:49:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1606 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:49:56 dist-test-slave-gmjp krb5kdc[579](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:49:56 dist-test-slave-gmjp krb5kdc[579](info): set up 2 sockets
Aug 06 02:49:56 dist-test-slave-gmjp krb5kdc[579](info): commencing operation
krb5kdc: starting...
W20260806 02:49:57.766395 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.210s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:49:57 dist-test-slave-gmjp krb5kdc[579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:49:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:49:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:49:57.897537 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46875
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:49:58.012071 595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:58.012449 595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:58.012539 595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:58.016690 595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:49:58.016772 595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:58.016809 595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:58.016861 595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:49:58.016896 595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:49:58.022394 595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:58.023499 595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:58.024634 595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:58.033412 602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:58.033404 604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:58.033404 606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:58.033720 595 server_base.cc:1034] running on GCE node
I20260806 02:49:58.034449 595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:58.035617 595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:58.036801 595 hybrid_clock.cc:648] HybridClock initialized: now 1785984598036786 us; error 33 us; skew 500 ppm
Aug 06 02:49:58 dist-test-slave-gmjp krb5kdc[579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:58.039602 595 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:49:58.040586 595 webserver.cc:460] Webserver started at http://127.0.0.1:33495/ using document root <none> and password file <none>
I20260806 02:49:58.041152 595 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:58.041242 595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:58.041471 595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:58.043519 595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "28fd9fcdf6d649eebc7725895bd752c4"
format_stamp: "Formatted at 2026-08-06 02:49:58 on dist-test-slave-gmjp"
server_key: "7c336eda8c7135e5cefd533dedd4ea89"
server_key_iv: "fa20e319930c34f865293a836780513c"
server_key_version: "encryptionkey@0"
I20260806 02:49:58.044090 595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "28fd9fcdf6d649eebc7725895bd752c4"
format_stamp: "Formatted at 2026-08-06 02:49:58 on dist-test-slave-gmjp"
server_key: "7c336eda8c7135e5cefd533dedd4ea89"
server_key_iv: "fa20e319930c34f865293a836780513c"
server_key_version: "encryptionkey@0"
I20260806 02:49:58.047655 595 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:49:58.050107 617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:58.051290 595 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 02:49:58.051427 595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "28fd9fcdf6d649eebc7725895bd752c4"
format_stamp: "Formatted at 2026-08-06 02:49:58 on dist-test-slave-gmjp"
server_key: "7c336eda8c7135e5cefd533dedd4ea89"
server_key_iv: "fa20e319930c34f865293a836780513c"
server_key_version: "encryptionkey@0"
I20260806 02:49:58.051538 595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:58.056058 595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:58.059369 595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:58.059569 595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:58.066934 595 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46875
I20260806 02:49:58.066952 689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46875 every 8 connection(s)
I20260806 02:49:58.067880 595 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:49:58.070777 690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:58.074254 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 595
I20260806 02:49:58.074390 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:49:58.074724 1758 external_mini_cluster.cc:1428] Setting key 561944f0a65b1fcfe4d77917c7fec0a3
I20260806 02:49:58.076920 690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28fd9fcdf6d649eebc7725895bd752c4: Bootstrap starting.
I20260806 02:49:58.079718 690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28fd9fcdf6d649eebc7725895bd752c4: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:58.080408 694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:58.076142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:47487 (local address 127.0.0.1:46875)
0806 02:49:58.076628 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:58.076646 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:58.076688 (+ 42us) server_negotiation.cc:407] Connection header received
0806 02:49:58.077500 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:58.077525 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:58.077870 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:58.078264 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:58.079002 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:58.079007 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:58.079023 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:49:58.079038 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:58.079108 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:49:58.079136 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:58.079239 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:58.079327 (+ 88us) server_negotiation.cc:299] Negotiation successful
0806 02:49:58.079477 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":122,"threads_started":1}
I20260806 02:49:58.080624 690 log.cc:824] T 00000000000000000000000000000000 P 28fd9fcdf6d649eebc7725895bd752c4: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:58.082819 690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28fd9fcdf6d649eebc7725895bd752c4: No bootstrap required, opened a new log
I20260806 02:49:58.083319 1758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:49:58.086784 690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28fd9fcdf6d649eebc7725895bd752c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28fd9fcdf6d649eebc7725895bd752c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46875 } }
I20260806 02:49:58.087006 694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:58.085374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:34955 (local address 127.0.0.1:46875)
0806 02:49:58.085510 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:58.085515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:58.085733 (+ 218us) server_negotiation.cc:407] Connection header received
0806 02:49:58.085818 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:58.085822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:58.085863 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:58.085963 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:58.086655 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:58.086659 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:58.086661 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:49:58.086672 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:58.086698 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:49:58.086711 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:58.086801 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:58.086857 (+ 56us) server_negotiation.cc:299] Negotiation successful
0806 02:49:58.086896 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:49:58.087059 690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28fd9fcdf6d649eebc7725895bd752c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:58.087110 690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28fd9fcdf6d649eebc7725895bd752c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28fd9fcdf6d649eebc7725895bd752c4, State: Initialized, Role: FOLLOWER
I20260806 02:49:58.087561 690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28fd9fcdf6d649eebc7725895bd752c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28fd9fcdf6d649eebc7725895bd752c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46875 } }
I20260806 02:49:58.087685 690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28fd9fcdf6d649eebc7725895bd752c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:49:58.087749 690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28fd9fcdf6d649eebc7725895bd752c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:49:58.087862 690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28fd9fcdf6d649eebc7725895bd752c4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:58.088825 690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28fd9fcdf6d649eebc7725895bd752c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28fd9fcdf6d649eebc7725895bd752c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46875 } }
I20260806 02:49:58.089156 690 leader_election.cc:302] T 00000000000000000000000000000000 P 28fd9fcdf6d649eebc7725895bd752c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28fd9fcdf6d649eebc7725895bd752c4; no voters:
I20260806 02:49:58.089442 690 leader_election.cc:288] T 00000000000000000000000000000000 P 28fd9fcdf6d649eebc7725895bd752c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:49:58.089564 704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28fd9fcdf6d649eebc7725895bd752c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:49:58.089811 704 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28fd9fcdf6d649eebc7725895bd752c4 [term 1 LEADER]: Becoming Leader. State: Replica: 28fd9fcdf6d649eebc7725895bd752c4, State: Running, Role: LEADER
I20260806 02:49:58.090111 704 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28fd9fcdf6d649eebc7725895bd752c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28fd9fcdf6d649eebc7725895bd752c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46875 } }
I20260806 02:49:58.090613 690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28fd9fcdf6d649eebc7725895bd752c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:49:58.091727 706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28fd9fcdf6d649eebc7725895bd752c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28fd9fcdf6d649eebc7725895bd752c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28fd9fcdf6d649eebc7725895bd752c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46875 } } }
I20260806 02:49:58.091866 706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28fd9fcdf6d649eebc7725895bd752c4 [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:58.092187 708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28fd9fcdf6d649eebc7725895bd752c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28fd9fcdf6d649eebc7725895bd752c4. Latest consensus state: current_term: 1 leader_uuid: "28fd9fcdf6d649eebc7725895bd752c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28fd9fcdf6d649eebc7725895bd752c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46875 } } }
I20260806 02:49:58.092347 708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28fd9fcdf6d649eebc7725895bd752c4 [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:58.092281 718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:49:58.095247 718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:49:58.100641 718 catalog_manager.cc:1347] Generated new cluster ID: 9f150f8c09364f829ed061dedf258ca9
I20260806 02:49:58.100711 718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:49:58.120044 718 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:49:58.120409 718 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:49:58.129168 718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28fd9fcdf6d649eebc7725895bd752c4: Generated new TSK 0
I20260806 02:49:58.129832 718 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:49:58.133231 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 595
2026-08-06T02:49:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1603 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:49:58 dist-test-slave-gmjp krb5kdc[737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:49:58 dist-test-slave-gmjp krb5kdc[737](info): set up 2 sockets
Aug 06 02:49:58 dist-test-slave-gmjp krb5kdc[737](info): commencing operation
krb5kdc: starting...
W20260806 02:49:59.385145 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.225s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:49:59 dist-test-slave-gmjp krb5kdc[737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:49:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:49:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:49:59.514932 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40947
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:49:59.625620 754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:49:59.625916 754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:49:59.626005 754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:49:59.630038 754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:49:59.630131 754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:49:59.630170 754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:49:59.630228 754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:49:59.630267 754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:49:59.635502 754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40947
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:49:59.636694 754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:49:59.637746 754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:49:59.646307 765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:49:59.646353 762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:59.646464 754 server_base.cc:1034] running on GCE node
W20260806 02:49:59.646335 761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:49:59.647274 754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:49:59.648452 754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:49:59.649639 754 hybrid_clock.cc:648] HybridClock initialized: now 1785984599649622 us; error 28 us; skew 500 ppm
Aug 06 02:49:59 dist-test-slave-gmjp krb5kdc[737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:49:59.657366 754 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:49:59.658290 754 webserver.cc:460] Webserver started at http://127.0.0.1:42575/ using document root <none> and password file <none>
I20260806 02:49:59.658883 754 fs_manager.cc:359] Metadata directory not provided
I20260806 02:49:59.658983 754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:49:59.659224 754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:49:59.661482 754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "5fce00820bd8437499b05843add731b4"
format_stamp: "Formatted at 2026-08-06 02:49:59 on dist-test-slave-gmjp"
server_key: "0fc48fb804656ce101ce87932c95b634"
server_key_iv: "6a514d25254f60564f6dd5b259312480"
server_key_version: "encryptionkey@0"
I20260806 02:49:59.662067 754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "5fce00820bd8437499b05843add731b4"
format_stamp: "Formatted at 2026-08-06 02:49:59 on dist-test-slave-gmjp"
server_key: "0fc48fb804656ce101ce87932c95b634"
server_key_iv: "6a514d25254f60564f6dd5b259312480"
server_key_version: "encryptionkey@0"
I20260806 02:49:59.665910 754 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:49:59.668447 774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:49:59.669680 754 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:49:59.669834 754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "5fce00820bd8437499b05843add731b4"
format_stamp: "Formatted at 2026-08-06 02:49:59 on dist-test-slave-gmjp"
server_key: "0fc48fb804656ce101ce87932c95b634"
server_key_iv: "6a514d25254f60564f6dd5b259312480"
server_key_version: "encryptionkey@0"
I20260806 02:49:59.669953 754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:49:59.680692 754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:49:59.684075 754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:49:59.684310 754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:49:59.691774 850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40947 every 8 connection(s)
I20260806 02:49:59.691797 754 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40947
I20260806 02:49:59.692903 754 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:49:59.695746 852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:49:59.700973 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 754
I20260806 02:49:59.701131 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:49:59.701432 1758 external_mini_cluster.cc:1428] Setting key 25eea5922e4f46cb2be4adb906bf9c1e
I20260806 02:49:59.701602 852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fce00820bd8437499b05843add731b4: Bootstrap starting.
I20260806 02:49:59.704311 852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fce00820bd8437499b05843add731b4: Neither blocks nor log segments found. Creating new log.
I20260806 02:49:59.705200 852 log.cc:824] T 00000000000000000000000000000000 P 5fce00820bd8437499b05843add731b4: Log is configured to *not* fsync() on all Append() calls
I20260806 02:49:59.707172 852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fce00820bd8437499b05843add731b4: No bootstrap required, opened a new log
Aug 06 02:49:59 dist-test-slave-gmjp krb5kdc[737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 02:49:59.711246 852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fce00820bd8437499b05843add731b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fce00820bd8437499b05843add731b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40947 } }
I20260806 02:49:59.711472 852 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fce00820bd8437499b05843add731b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:49:59.711532 852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fce00820bd8437499b05843add731b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fce00820bd8437499b05843add731b4, State: Initialized, Role: FOLLOWER
I20260806 02:49:59.712087 852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fce00820bd8437499b05843add731b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fce00820bd8437499b05843add731b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40947 } }
I20260806 02:49:59.712327 852 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fce00820bd8437499b05843add731b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:49:59.712409 852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fce00820bd8437499b05843add731b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:49:59.712524 852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fce00820bd8437499b05843add731b4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:49:59.713467 852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fce00820bd8437499b05843add731b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fce00820bd8437499b05843add731b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40947 } }
I20260806 02:49:59.713826 852 leader_election.cc:302] T 00000000000000000000000000000000 P 5fce00820bd8437499b05843add731b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fce00820bd8437499b05843add731b4; no voters:
I20260806 02:49:59.714192 852 leader_election.cc:288] T 00000000000000000000000000000000 P 5fce00820bd8437499b05843add731b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:49:59.714283 858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fce00820bd8437499b05843add731b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:49:59.714516 858 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fce00820bd8437499b05843add731b4 [term 1 LEADER]: Becoming Leader. State: Replica: 5fce00820bd8437499b05843add731b4, State: Running, Role: LEADER
I20260806 02:49:59.714818 858 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fce00820bd8437499b05843add731b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fce00820bd8437499b05843add731b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40947 } }
I20260806 02:49:59.715600 852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fce00820bd8437499b05843add731b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:49:59.716822 862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fce00820bd8437499b05843add731b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fce00820bd8437499b05843add731b4. Latest consensus state: current_term: 1 leader_uuid: "5fce00820bd8437499b05843add731b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fce00820bd8437499b05843add731b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40947 } } }
I20260806 02:49:59.716948 855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:59.702910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:52367 (local address 127.0.0.1:40947)
0806 02:49:59.703380 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:59.703396 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:59.703440 (+ 44us) server_negotiation.cc:407] Connection header received
0806 02:49:59.704103 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:59.704120 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:59.704600 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:59.704883 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:59.705885 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:59.706860 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:59.707810 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:59.708352 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:59.708551 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:59.711397 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:59.711407 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:59.711422 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:59.711471 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:59.713343 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:59.713891 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:59.713897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:59.713904 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:59.714095 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:59.714575 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:59.714578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:59.714580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:59.715066 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:59.715278 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:59.715645 (+ 367us) server_negotiation.cc:299] Negotiation successful
0806 02:49:59.715881 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":158,"threads_started":1}
I20260806 02:49:59.716995 862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fce00820bd8437499b05843add731b4 [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:59.716830 859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fce00820bd8437499b05843add731b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fce00820bd8437499b05843add731b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fce00820bd8437499b05843add731b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40947 } } }
I20260806 02:49:59.717314 859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fce00820bd8437499b05843add731b4 [sys.catalog]: This master's current role is: LEADER
I20260806 02:49:59.717417 870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:49:59.720328 1758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:49:59.720649 870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:49:59.726732 870 catalog_manager.cc:1347] Generated new cluster ID: 4dcfa2c2cec74c2996a20d0115bb8720
I20260806 02:49:59.726804 870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:49:59.731510 855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:49:59.722516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:56909 (local address 127.0.0.1:40947)
0806 02:49:59.722626 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0806 02:49:59.722631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:49:59.722977 (+ 346us) server_negotiation.cc:407] Connection header received
0806 02:49:59.723084 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:49:59.723087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:49:59.723133 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:49:59.723216 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:49:59.724083 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:59.724829 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:59.725783 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:49:59.726032 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:49:59.726141 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:49:59.727933 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:49:59.727938 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:49:59.727941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:49:59.727968 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:49:59.729645 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:59.730173 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:59.730177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:59.730179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:59.730271 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:49:59.730604 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:49:59.730610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:49:59.730612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:49:59.730828 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:49:59.730991 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 02:49:59.731243 (+ 252us) server_negotiation.cc:299] Negotiation successful
0806 02:49:59.731343 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 02:49:59.768725 870 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:49:59.769769 870 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:49:59.774600 870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fce00820bd8437499b05843add731b4: Generated new TSK 0
I20260806 02:49:59.775318 870 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:49:59.780927 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 754
W20260806 02:49:59.794488 733 connection.cc:535] client connection to 127.0.0.1:40947 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40947): Connection reset by peer (error 104)
2026-08-06T02:49:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1648 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:49:59 dist-test-slave-gmjp krb5kdc[889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:49:59 dist-test-slave-gmjp krb5kdc[889](info): set up 2 sockets
Aug 06 02:49:59 dist-test-slave-gmjp krb5kdc[889](info): commencing operation
krb5kdc: starting...
W20260806 02:50:01.003070 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.194s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:50:01 dist-test-slave-gmjp krb5kdc[889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:50:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:50:01.134734 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46069
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:01.252097 907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:01.252446 907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:01.252527 907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:01.256629 907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:01.256719 907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:01.256757 907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:01.256816 907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:01.256856 907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:01.262286 907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46069
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:01.263446 907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:01.264578 907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:01.273198 912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:01.273198 916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:01.273227 913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:01.273478 907 server_base.cc:1034] running on GCE node
I20260806 02:50:01.274197 907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:01.275300 907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:01.276489 907 hybrid_clock.cc:648] HybridClock initialized: now 1785984601276474 us; error 24 us; skew 500 ppm
Aug 06 02:50:01 dist-test-slave-gmjp krb5kdc[889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:01.279403 907 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:50:01.280447 907 webserver.cc:460] Webserver started at http://127.0.0.1:33025/ using document root <none> and password file <none>
I20260806 02:50:01.281126 907 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:01.281234 907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:01.281487 907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:01.283744 907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "5d600c18a9384dfbbfb1279f81312cbb"
format_stamp: "Formatted at 2026-08-06 02:50:01 on dist-test-slave-gmjp"
server_key: "6343c42a52be64ed1220aa43874bf82c"
server_key_iv: "50da175660d864e74a601c188cd09bc5"
server_key_version: "encryptionkey@0"
I20260806 02:50:01.284425 907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "5d600c18a9384dfbbfb1279f81312cbb"
format_stamp: "Formatted at 2026-08-06 02:50:01 on dist-test-slave-gmjp"
server_key: "6343c42a52be64ed1220aa43874bf82c"
server_key_iv: "50da175660d864e74a601c188cd09bc5"
server_key_version: "encryptionkey@0"
I20260806 02:50:01.288545 907 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:50:01.291074 927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:01.292470 907 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:50:01.292582 907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "5d600c18a9384dfbbfb1279f81312cbb"
format_stamp: "Formatted at 2026-08-06 02:50:01 on dist-test-slave-gmjp"
server_key: "6343c42a52be64ed1220aa43874bf82c"
server_key_iv: "50da175660d864e74a601c188cd09bc5"
server_key_version: "encryptionkey@0"
I20260806 02:50:01.292667 907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:01.307616 907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:01.310747 907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:01.310920 907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:01.318181 907 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46069
I20260806 02:50:01.318190 1002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46069 every 8 connection(s)
I20260806 02:50:01.319167 907 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:01.321848 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 907
I20260806 02:50:01.321993 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:50:01.322095 1003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:01.322376 1758 external_mini_cluster.cc:1428] Setting key 4969ee0078944ec7380a8069ad61d206
I20260806 02:50:01.329007 1003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d600c18a9384dfbbfb1279f81312cbb: Bootstrap starting.
I20260806 02:50:01.331452 1003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d600c18a9384dfbbfb1279f81312cbb: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:01.331596 1008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:01.323712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:53089 (local address 127.0.0.1:46069)
0806 02:50:01.324134 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:01.324196 (+ 62us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:01.324239 (+ 43us) server_negotiation.cc:407] Connection header received
0806 02:50:01.324860 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:01.324875 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:01.325207 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:01.325482 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:01.325871 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:01.326876 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:01.328185 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:01.328652 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:01.328798 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:01.329879 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:01.329884 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:01.329896 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:01.330277 (+ 381us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:01.330333 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:01.330356 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:01.330457 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:01.330695 (+ 238us) server_negotiation.cc:299] Negotiation successful
0806 02:50:01.330839 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":155,"threads_started":1}
I20260806 02:50:01.332332 1003 log.cc:824] T 00000000000000000000000000000000 P 5d600c18a9384dfbbfb1279f81312cbb: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:01.334328 1003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d600c18a9384dfbbfb1279f81312cbb: No bootstrap required, opened a new log
I20260806 02:50:01.334513 1758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:50:01.339125 1003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d600c18a9384dfbbfb1279f81312cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d600c18a9384dfbbfb1279f81312cbb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46069 } }
I20260806 02:50:01.339437 1003 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d600c18a9384dfbbfb1279f81312cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:01.339524 1003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d600c18a9384dfbbfb1279f81312cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d600c18a9384dfbbfb1279f81312cbb, State: Initialized, Role: FOLLOWER
I20260806 02:50:01.340037 1003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d600c18a9384dfbbfb1279f81312cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d600c18a9384dfbbfb1279f81312cbb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46069 } }
I20260806 02:50:01.340241 1003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d600c18a9384dfbbfb1279f81312cbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:01.340312 1003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d600c18a9384dfbbfb1279f81312cbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:01.340435 1003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d600c18a9384dfbbfb1279f81312cbb [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:01.341353 1008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:01.336805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:54477 (local address 127.0.0.1:46069)
0806 02:50:01.336933 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:01.336938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:01.337204 (+ 266us) server_negotiation.cc:407] Connection header received
0806 02:50:01.337317 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:01.337320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:01.337358 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:01.337421 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:01.337715 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:01.338196 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:01.339311 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:01.339673 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:01.339780 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:01.340771 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:01.340777 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:01.340779 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:01.340793 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:01.340820 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:01.340834 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:01.340921 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:01.341181 (+ 260us) server_negotiation.cc:299] Negotiation successful
0806 02:50:01.341219 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 02:50:01.341879 1003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d600c18a9384dfbbfb1279f81312cbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d600c18a9384dfbbfb1279f81312cbb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46069 } }
I20260806 02:50:01.342260 1003 leader_election.cc:302] T 00000000000000000000000000000000 P 5d600c18a9384dfbbfb1279f81312cbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d600c18a9384dfbbfb1279f81312cbb; no voters:
I20260806 02:50:01.342695 1003 leader_election.cc:288] T 00000000000000000000000000000000 P 5d600c18a9384dfbbfb1279f81312cbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:01.342767 1019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d600c18a9384dfbbfb1279f81312cbb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:01.343022 1019 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d600c18a9384dfbbfb1279f81312cbb [term 1 LEADER]: Becoming Leader. State: Replica: 5d600c18a9384dfbbfb1279f81312cbb, State: Running, Role: LEADER
I20260806 02:50:01.343330 1019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d600c18a9384dfbbfb1279f81312cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d600c18a9384dfbbfb1279f81312cbb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46069 } }
I20260806 02:50:01.344122 1003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d600c18a9384dfbbfb1279f81312cbb [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:01.345237 1021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d600c18a9384dfbbfb1279f81312cbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d600c18a9384dfbbfb1279f81312cbb. Latest consensus state: current_term: 1 leader_uuid: "5d600c18a9384dfbbfb1279f81312cbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d600c18a9384dfbbfb1279f81312cbb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46069 } } }
I20260806 02:50:01.345405 1021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d600c18a9384dfbbfb1279f81312cbb [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:01.345247 1022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d600c18a9384dfbbfb1279f81312cbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d600c18a9384dfbbfb1279f81312cbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d600c18a9384dfbbfb1279f81312cbb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46069 } } }
I20260806 02:50:01.345597 1022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d600c18a9384dfbbfb1279f81312cbb [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:01.346712 1036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:01.349700 1036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:01.355381 1036 catalog_manager.cc:1347] Generated new cluster ID: 2f1f272d95764f9199df97ca786424cc
I20260806 02:50:01.355463 1036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:01.374399 1036 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:01.375860 1036 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:01.388036 1036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d600c18a9384dfbbfb1279f81312cbb: Generated new TSK 0
I20260806 02:50:01.388839 1036 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:50:01.402275 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 907
W20260806 02:50:01.418547 887 connection.cc:535] client connection to 127.0.0.1:46069 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46069): Connection reset by peer (error 104)
2026-08-06T02:50:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1624 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:50:01 dist-test-slave-gmjp krb5kdc[1055](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:50:01 dist-test-slave-gmjp krb5kdc[1055](info): set up 2 sockets
Aug 06 02:50:01 dist-test-slave-gmjp krb5kdc[1055](info): commencing operation
krb5kdc: starting...
W20260806 02:50:02.656664 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.223s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:50:02 dist-test-slave-gmjp krb5kdc[1055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:50:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:50:02.789369 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36155
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:02.906004 1075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:02.906296 1075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:02.906340 1075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:02.910440 1075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:02.910501 1075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:02.910521 1075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:02.910588 1075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:02.910621 1075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:02.916014 1075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36155
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984313887056-1758-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:02.917234 1075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:02.918295 1075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:02.927055 1082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:02.927099 1085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:02.927270 1075 server_base.cc:1034] running on GCE node
W20260806 02:50:02.927086 1081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:02.927892 1075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:02.929159 1075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:02.930330 1075 hybrid_clock.cc:648] HybridClock initialized: now 1785984602930309 us; error 42 us; skew 500 ppm
Aug 06 02:50:02 dist-test-slave-gmjp krb5kdc[1055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:02.933503 1075 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:50:02.934415 1075 webserver.cc:460] Webserver started at http://127.0.0.1:46473/ using document root <none> and password file <none>
I20260806 02:50:02.934996 1075 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:02.935092 1075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:02.935341 1075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:02.937556 1075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "21a272753cba47ae815723e54470409d"
format_stamp: "Formatted at 2026-08-06 02:50:02 on dist-test-slave-gmjp"
server_key: "f99e721e5b704c046f12abbce7501aea"
server_key_iv: "56119ec4c44e274ec7dc4b52021d327a"
server_key_version: "encryptionkey@0"
I20260806 02:50:02.938141 1075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "21a272753cba47ae815723e54470409d"
format_stamp: "Formatted at 2026-08-06 02:50:02 on dist-test-slave-gmjp"
server_key: "f99e721e5b704c046f12abbce7501aea"
server_key_iv: "56119ec4c44e274ec7dc4b52021d327a"
server_key_version: "encryptionkey@0"
I20260806 02:50:02.941963 1075 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:50:02.944470 1093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:02.945735 1075 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 02:50:02.945886 1075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "21a272753cba47ae815723e54470409d"
format_stamp: "Formatted at 2026-08-06 02:50:02 on dist-test-slave-gmjp"
server_key: "f99e721e5b704c046f12abbce7501aea"
server_key_iv: "56119ec4c44e274ec7dc4b52021d327a"
server_key_version: "encryptionkey@0"
I20260806 02:50:02.946002 1075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:02.950646 1075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:02.954002 1075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:02.954216 1075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:02.961426 1075 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36155
I20260806 02:50:02.961483 1175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36155 every 8 connection(s)
I20260806 02:50:02.962483 1075 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:02.965539 1177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:02.967348 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 1075
I20260806 02:50:02.967502 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:50:02.967804 1758 external_mini_cluster.cc:1428] Setting key d3b45834715a662e45388196cd7a30c0
I20260806 02:50:02.972988 1177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21a272753cba47ae815723e54470409d: Bootstrap starting.
I20260806 02:50:02.973980 1181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:02.969501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:56329 (local address 127.0.0.1:36155)
0806 02:50:02.969982 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:02.970002 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:02.970046 (+ 44us) server_negotiation.cc:407] Connection header received
0806 02:50:02.970916 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:02.970944 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:02.971323 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:02.971716 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:02.972555 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:02.972561 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:02.972576 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:02.972588 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:02.972655 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:02.972684 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:02.972832 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:02.972914 (+ 82us) server_negotiation.cc:299] Negotiation successful
0806 02:50:02.973077 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":140,"threads_started":1}
I20260806 02:50:02.975560 1177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21a272753cba47ae815723e54470409d: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:02.976437 1177 log.cc:824] T 00000000000000000000000000000000 P 21a272753cba47ae815723e54470409d: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:02.977006 1758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:50:02.978571 1177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21a272753cba47ae815723e54470409d: No bootstrap required, opened a new log
I20260806 02:50:02.980968 1181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:02.979097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.87:55591 (local address 127.0.0.1:36155)
0806 02:50:02.979223 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:02.979228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:02.979482 (+ 254us) server_negotiation.cc:407] Connection header received
0806 02:50:02.979609 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:02.979612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:02.979643 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:02.979713 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:02.980537 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:02.980540 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:02.980542 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:02.980553 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:02.980576 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:02.980587 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:02.980649 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:02.980763 (+ 114us) server_negotiation.cc:299] Negotiation successful
0806 02:50:02.980810 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:50:02.982887 1177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21a272753cba47ae815723e54470409d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21a272753cba47ae815723e54470409d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36155 } }
I20260806 02:50:02.983130 1177 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21a272753cba47ae815723e54470409d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:02.983188 1177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21a272753cba47ae815723e54470409d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21a272753cba47ae815723e54470409d, State: Initialized, Role: FOLLOWER
I20260806 02:50:02.983676 1177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21a272753cba47ae815723e54470409d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21a272753cba47ae815723e54470409d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36155 } }
I20260806 02:50:02.983821 1177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21a272753cba47ae815723e54470409d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:02.983871 1177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21a272753cba47ae815723e54470409d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:02.983945 1177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21a272753cba47ae815723e54470409d [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:02.984863 1177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21a272753cba47ae815723e54470409d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21a272753cba47ae815723e54470409d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36155 } }
I20260806 02:50:02.985152 1177 leader_election.cc:302] T 00000000000000000000000000000000 P 21a272753cba47ae815723e54470409d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21a272753cba47ae815723e54470409d; no voters:
I20260806 02:50:02.985416 1177 leader_election.cc:288] T 00000000000000000000000000000000 P 21a272753cba47ae815723e54470409d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:02.985525 1194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21a272753cba47ae815723e54470409d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:02.985761 1194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21a272753cba47ae815723e54470409d [term 1 LEADER]: Becoming Leader. State: Replica: 21a272753cba47ae815723e54470409d, State: Running, Role: LEADER
I20260806 02:50:02.986124 1194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21a272753cba47ae815723e54470409d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21a272753cba47ae815723e54470409d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36155 } }
I20260806 02:50:02.986670 1177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21a272753cba47ae815723e54470409d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:02.987689 1197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21a272753cba47ae815723e54470409d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21a272753cba47ae815723e54470409d. Latest consensus state: current_term: 1 leader_uuid: "21a272753cba47ae815723e54470409d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21a272753cba47ae815723e54470409d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36155 } } }
I20260806 02:50:02.987823 1197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21a272753cba47ae815723e54470409d [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:02.988210 1208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:02.988485 1196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21a272753cba47ae815723e54470409d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21a272753cba47ae815723e54470409d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21a272753cba47ae815723e54470409d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36155 } } }
I20260806 02:50:02.988663 1196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21a272753cba47ae815723e54470409d [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:02.991428 1208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:02.997229 1208 catalog_manager.cc:1347] Generated new cluster ID: 383e85e75ce94e6c852281b334146396
I20260806 02:50:02.997328 1208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:03.012951 1208 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:03.013375 1208 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:03.019022 1208 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21a272753cba47ae815723e54470409d: Generated new TSK 0
I20260806 02:50:03.019907 1208 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:50:03.033205 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 1075
2026-08-06T02:50:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1630 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:50:03 dist-test-slave-gmjp krb5kdc[1226](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:50:03 dist-test-slave-gmjp krb5kdc[1226](info): set up 2 sockets
Aug 06 02:50:03 dist-test-slave-gmjp krb5kdc[1226](info): commencing operation
krb5kdc: starting...
W20260806 02:50:04.258846 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.195s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:50:04 dist-test-slave-gmjp krb5kdc[1226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:50:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
I20260806 02:50:04.392565 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:37423
--webserver_interface=127.1.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:34091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:37423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:04.510264 1247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:04.510565 1247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:04.510651 1247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:04.514659 1247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:04.514748 1247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:04.514787 1247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:04.514842 1247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:04.514875 1247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:04.520239 1247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:34091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:37423
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:37423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:04.521371 1247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:04.522421 1247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:04.531273 1253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:04.531491 1247 server_base.cc:1034] running on GCE node
W20260806 02:50:04.531334 1254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:04.531283 1258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:04.532323 1247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:04.533497 1247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:04.534689 1247 hybrid_clock.cc:648] HybridClock initialized: now 1785984604534676 us; error 24 us; skew 500 ppm
Aug 06 02:50:04 dist-test-slave-gmjp krb5kdc[1226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984604, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:04.537573 1247 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:50:04.538473 1247 webserver.cc:460] Webserver started at http://127.1.183.190:46187/ using document root <none> and password file <none>
I20260806 02:50:04.539045 1247 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:04.539140 1247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:04.539412 1247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:04.541556 1247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "fa7c65f8919b483cbd7384bf684e1fc9"
format_stamp: "Formatted at 2026-08-06 02:50:04 on dist-test-slave-gmjp"
server_key: "d1cb932b80e4a480008d93d0d80fa31f"
server_key_iv: "0c4d87cfca43565718e635e55b939c85"
server_key_version: "encryptionkey@0"
I20260806 02:50:04.542135 1247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "fa7c65f8919b483cbd7384bf684e1fc9"
format_stamp: "Formatted at 2026-08-06 02:50:04 on dist-test-slave-gmjp"
server_key: "d1cb932b80e4a480008d93d0d80fa31f"
server_key_iv: "0c4d87cfca43565718e635e55b939c85"
server_key_version: "encryptionkey@0"
I20260806 02:50:04.545871 1247 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260806 02:50:04.548389 1267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:04.549768 1247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:50:04.549917 1247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "fa7c65f8919b483cbd7384bf684e1fc9"
format_stamp: "Formatted at 2026-08-06 02:50:04 on dist-test-slave-gmjp"
server_key: "d1cb932b80e4a480008d93d0d80fa31f"
server_key_iv: "0c4d87cfca43565718e635e55b939c85"
server_key_version: "encryptionkey@0"
I20260806 02:50:04.550033 1247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:04.564584 1247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:04.567656 1247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:04.567891 1247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:04.575295 1247 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:37423
I20260806 02:50:04.575325 1338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:37423 every 8 connection(s)
I20260806 02:50:04.576294 1247 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:04.579095 1340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:04.580346 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 1247
I20260806 02:50:04.580502 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:50:04.580793 1758 external_mini_cluster.cc:1428] Setting key fbe1b901aace8eaa2aa7b9faf2258935
I20260806 02:50:04.585289 1340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa7c65f8919b483cbd7384bf684e1fc9: Bootstrap starting.
I20260806 02:50:04.587525 1340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa7c65f8919b483cbd7384bf684e1fc9: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:04.588449 1340 log.cc:824] T 00000000000000000000000000000000 P fa7c65f8919b483cbd7384bf684e1fc9: Log is configured to *not* fsync() on all Append() calls
Aug 06 02:50:04 dist-test-slave-gmjp krb5kdc[1226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:50:04.590428 1340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa7c65f8919b483cbd7384bf684e1fc9: No bootstrap required, opened a new log
I20260806 02:50:04.594095 1340 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa7c65f8919b483cbd7384bf684e1fc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa7c65f8919b483cbd7384bf684e1fc9" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 37423 } }
I20260806 02:50:04.594375 1340 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa7c65f8919b483cbd7384bf684e1fc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:04.594445 1340 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa7c65f8919b483cbd7384bf684e1fc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa7c65f8919b483cbd7384bf684e1fc9, State: Initialized, Role: FOLLOWER
I20260806 02:50:04.594789 1345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:04.582218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56979 (local address 127.1.183.190:37423)
0806 02:50:04.582658 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:04.582673 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:04.582707 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:50:04.583359 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:04.583376 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:04.583715 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:04.584046 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:04.584919 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:04.585965 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:04.586840 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:04.587294 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:04.587465 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:04.590075 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:04.590081 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:04.590093 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:04.590119 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:04.591787 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:04.592355 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:04.592360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:04.592364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:04.592484 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:04.592836 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:04.592839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:04.592840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:04.593206 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:04.593459 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:04.593714 (+ 255us) server_negotiation.cc:299] Negotiation successful
0806 02:50:04.593931 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":141,"threads_started":1}
I20260806 02:50:04.594959 1340 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa7c65f8919b483cbd7384bf684e1fc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa7c65f8919b483cbd7384bf684e1fc9" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 37423 } }
I20260806 02:50:04.595124 1340 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa7c65f8919b483cbd7384bf684e1fc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:04.595193 1340 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa7c65f8919b483cbd7384bf684e1fc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:04.595288 1340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa7c65f8919b483cbd7384bf684e1fc9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:04.596406 1340 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa7c65f8919b483cbd7384bf684e1fc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa7c65f8919b483cbd7384bf684e1fc9" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 37423 } }
I20260806 02:50:04.596786 1340 leader_election.cc:302] T 00000000000000000000000000000000 P fa7c65f8919b483cbd7384bf684e1fc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa7c65f8919b483cbd7384bf684e1fc9; no voters:
I20260806 02:50:04.597138 1340 leader_election.cc:288] T 00000000000000000000000000000000 P fa7c65f8919b483cbd7384bf684e1fc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:04.597214 1347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa7c65f8919b483cbd7384bf684e1fc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:04.597394 1347 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa7c65f8919b483cbd7384bf684e1fc9 [term 1 LEADER]: Becoming Leader. State: Replica: fa7c65f8919b483cbd7384bf684e1fc9, State: Running, Role: LEADER
I20260806 02:50:04.598038 1758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:50:04.598170 1347 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa7c65f8919b483cbd7384bf684e1fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa7c65f8919b483cbd7384bf684e1fc9" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 37423 } }
I20260806 02:50:04.598481 1340 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa7c65f8919b483cbd7384bf684e1fc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:04.600431 1348 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa7c65f8919b483cbd7384bf684e1fc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa7c65f8919b483cbd7384bf684e1fc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa7c65f8919b483cbd7384bf684e1fc9" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 37423 } } }
I20260806 02:50:04.600564 1348 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa7c65f8919b483cbd7384bf684e1fc9 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:04.600925 1349 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa7c65f8919b483cbd7384bf684e1fc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa7c65f8919b483cbd7384bf684e1fc9. Latest consensus state: current_term: 1 leader_uuid: "fa7c65f8919b483cbd7384bf684e1fc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa7c65f8919b483cbd7384bf684e1fc9" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 37423 } } }
I20260806 02:50:04.601032 1349 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa7c65f8919b483cbd7384bf684e1fc9 [sys.catalog]: This master's current role is: LEADER
W20260806 02:50:04.603559 1373 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fa7c65f8919b483cbd7384bf684e1fc9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:50:04.603662 1373 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:50:04.603794 1376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:04.604626 1376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:04.610407 1345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:04.600501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37103 (local address 127.1.183.190:37423)
0806 02:50:04.600629 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:04.600634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:04.600969 (+ 335us) server_negotiation.cc:407] Connection header received
0806 02:50:04.601188 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:04.601191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:04.601239 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:04.601325 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:04.602162 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:04.602910 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:04.604122 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:04.604501 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:04.604610 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:04.606378 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:04.606382 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:04.606385 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:04.606421 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:04.608348 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:04.608917 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:04.608922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:04.608924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:04.609003 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:04.609447 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:04.609451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:04.609452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:04.609674 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:04.609871 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:04.610129 (+ 258us) server_negotiation.cc:299] Negotiation successful
0806 02:50:04.610230 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:50:04.611995 1376 catalog_manager.cc:1347] Generated new cluster ID: d05f943beefe4f0db53b4c7407cda75a
I20260806 02:50:04.612073 1376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:04.624755 1376 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:04.625773 1376 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:04.632216 1376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa7c65f8919b483cbd7384bf684e1fc9: Generated new TSK 0
I20260806 02:50:04.632957 1376 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:50:04.640538 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 1247
W20260806 02:50:04.651878 1223 connection.cc:535] client connection to 127.1.183.190:37423 recv error: Network error: failed to read from TLS socket (remote: 127.1.183.190:37423): Connection reset by peer (error 104)
2026-08-06T02:50:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1602 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:50:05 dist-test-slave-gmjp krb5kdc[1383](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:50:05 dist-test-slave-gmjp krb5kdc[1383](info): set up 2 sockets
Aug 06 02:50:05 dist-test-slave-gmjp krb5kdc[1383](info): commencing operation
krb5kdc: starting...
W20260806 02:50:06.650524 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.984s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:50:06 dist-test-slave-gmjp krb5kdc[1383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:50:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
I20260806 02:50:06.783114 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:36659
--webserver_interface=127.1.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:37691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:36659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:06.896951 1405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:06.897275 1405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:06.897353 1405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:06.901435 1405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:06.901527 1405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:06.901564 1405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:06.901623 1405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:06.901661 1405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:06.907022 1405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:37691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:36659
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:36659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:06.908242 1405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:06.909330 1405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:06.917979 1414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:06.917973 1418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:06.917973 1416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:06.918344 1405 server_base.cc:1034] running on GCE node
I20260806 02:50:06.918891 1405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:06.919983 1405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:06.921169 1405 hybrid_clock.cc:648] HybridClock initialized: now 1785984606921162 us; error 51 us; skew 500 ppm
Aug 06 02:50:06 dist-test-slave-gmjp krb5kdc[1383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984606, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:06.923859 1405 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:50:06.924803 1405 webserver.cc:460] Webserver started at http://127.1.183.190:34333/ using document root <none> and password file <none>
I20260806 02:50:06.925371 1405 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:06.925464 1405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:06.925699 1405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:06.927743 1405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "43a780e2091f41f8aee409861db17da1"
format_stamp: "Formatted at 2026-08-06 02:50:06 on dist-test-slave-gmjp"
server_key: "31da11e5fb6c60097ec1acbf94be08d6"
server_key_iv: "1d60b180493e8064811c8da8fe66548d"
server_key_version: "encryptionkey@0"
I20260806 02:50:06.928398 1405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "43a780e2091f41f8aee409861db17da1"
format_stamp: "Formatted at 2026-08-06 02:50:06 on dist-test-slave-gmjp"
server_key: "31da11e5fb6c60097ec1acbf94be08d6"
server_key_iv: "1d60b180493e8064811c8da8fe66548d"
server_key_version: "encryptionkey@0"
I20260806 02:50:06.932082 1405 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:50:06.934530 1428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:06.935782 1405 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 02:50:06.935889 1405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "43a780e2091f41f8aee409861db17da1"
format_stamp: "Formatted at 2026-08-06 02:50:06 on dist-test-slave-gmjp"
server_key: "31da11e5fb6c60097ec1acbf94be08d6"
server_key_iv: "1d60b180493e8064811c8da8fe66548d"
server_key_version: "encryptionkey@0"
I20260806 02:50:06.935990 1405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:06.948102 1405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:06.951298 1405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:06.951532 1405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:06.958446 1405 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:36659
I20260806 02:50:06.958523 1507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:36659 every 8 connection(s)
I20260806 02:50:06.959507 1405 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:06.960024 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 1405
I20260806 02:50:06.960216 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:50:06.960536 1758 external_mini_cluster.cc:1428] Setting key 1bf03bcfd1464a2354eb8695be9422fc
I20260806 02:50:06.962862 1509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:06.968529 1509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43a780e2091f41f8aee409861db17da1: Bootstrap starting.
I20260806 02:50:06.969720 1518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:06.961927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51623 (local address 127.1.183.190:36659)
0806 02:50:06.962395 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:06.962414 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:06.962449 (+ 35us) server_negotiation.cc:407] Connection header received
0806 02:50:06.963153 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:06.963170 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:06.963507 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:06.963840 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:06.964312 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:06.965315 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:06.966295 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:06.966791 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:06.966973 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:06.967916 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:06.967921 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:06.967934 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:06.968375 (+ 441us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:06.968433 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:06.968457 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:06.968596 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:06.968899 (+ 303us) server_negotiation.cc:299] Negotiation successful
0806 02:50:06.969014 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":159,"threads_started":1}
I20260806 02:50:06.971077 1509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43a780e2091f41f8aee409861db17da1: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:06.971954 1509 log.cc:824] T 00000000000000000000000000000000 P 43a780e2091f41f8aee409861db17da1: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:06.972829 1758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:50:06.974026 1509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43a780e2091f41f8aee409861db17da1: No bootstrap required, opened a new log
I20260806 02:50:06.978788 1509 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43a780e2091f41f8aee409861db17da1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43a780e2091f41f8aee409861db17da1" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 36659 } }
I20260806 02:50:06.979120 1509 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43a780e2091f41f8aee409861db17da1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:06.979188 1509 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43a780e2091f41f8aee409861db17da1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43a780e2091f41f8aee409861db17da1, State: Initialized, Role: FOLLOWER
I20260806 02:50:06.979717 1509 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43a780e2091f41f8aee409861db17da1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43a780e2091f41f8aee409861db17da1" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 36659 } }
I20260806 02:50:06.979842 1518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:06.975190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46187 (local address 127.1.183.190:36659)
0806 02:50:06.975319 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:06.975323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:06.975638 (+ 315us) server_negotiation.cc:407] Connection header received
0806 02:50:06.975771 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:06.975773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:06.975816 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:06.975883 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:06.976245 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:06.976745 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:06.977932 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:06.978164 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:06.978257 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:06.979219 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:06.979223 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:06.979231 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:06.979261 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:06.979290 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:06.979303 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:06.979377 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:06.979564 (+ 187us) server_negotiation.cc:299] Negotiation successful
0806 02:50:06.979630 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:50:06.979856 1509 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43a780e2091f41f8aee409861db17da1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:06.979952 1509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43a780e2091f41f8aee409861db17da1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:06.980047 1509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43a780e2091f41f8aee409861db17da1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:06.981032 1509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43a780e2091f41f8aee409861db17da1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43a780e2091f41f8aee409861db17da1" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 36659 } }
I20260806 02:50:06.981369 1509 leader_election.cc:302] T 00000000000000000000000000000000 P 43a780e2091f41f8aee409861db17da1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43a780e2091f41f8aee409861db17da1; no voters:
I20260806 02:50:06.981707 1509 leader_election.cc:288] T 00000000000000000000000000000000 P 43a780e2091f41f8aee409861db17da1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:06.981812 1528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43a780e2091f41f8aee409861db17da1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:06.982065 1528 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43a780e2091f41f8aee409861db17da1 [term 1 LEADER]: Becoming Leader. State: Replica: 43a780e2091f41f8aee409861db17da1, State: Running, Role: LEADER
I20260806 02:50:06.982357 1528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43a780e2091f41f8aee409861db17da1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43a780e2091f41f8aee409861db17da1" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 36659 } }
I20260806 02:50:06.982933 1509 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43a780e2091f41f8aee409861db17da1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:06.984113 1531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43a780e2091f41f8aee409861db17da1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43a780e2091f41f8aee409861db17da1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43a780e2091f41f8aee409861db17da1" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 36659 } } }
I20260806 02:50:06.984278 1531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43a780e2091f41f8aee409861db17da1 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:06.984758 1532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43a780e2091f41f8aee409861db17da1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43a780e2091f41f8aee409861db17da1. Latest consensus state: current_term: 1 leader_uuid: "43a780e2091f41f8aee409861db17da1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43a780e2091f41f8aee409861db17da1" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 36659 } } }
I20260806 02:50:06.984858 1532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43a780e2091f41f8aee409861db17da1 [sys.catalog]: This master's current role is: LEADER
W20260806 02:50:06.987756 1554 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 43a780e2091f41f8aee409861db17da1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:50:06.987823 1554 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:50:06.987914 1543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:06.988664 1543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:06.993971 1543 catalog_manager.cc:1347] Generated new cluster ID: b2e15491fa17486fb7e03b9e34780573
I20260806 02:50:06.994033 1543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:07.004094 1543 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:07.004978 1543 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:07.013402 1543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43a780e2091f41f8aee409861db17da1: Generated new TSK 0
I20260806 02:50:07.014300 1543 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:50:07.028527 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 1405
W20260806 02:50:07.038522 1381 connection.cc:535] client connection to 127.1.183.190:36659 recv error: Network error: failed to read from TLS socket (remote: 127.1.183.190:36659): Connection reset by peer (error 104)
2026-08-06T02:50:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2386 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:50:07 dist-test-slave-gmjp krb5kdc[1561](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:50:07 dist-test-slave-gmjp krb5kdc[1561](info): set up 2 sockets
Aug 06 02:50:07 dist-test-slave-gmjp krb5kdc[1561](info): commencing operation
krb5kdc: starting...
W20260806 02:50:08.282107 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.229s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:50:08 dist-test-slave-gmjp krb5kdc[1561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:50:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
I20260806 02:50:08.419286 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:38817
--webserver_interface=127.1.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:36591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:38817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:08.541638 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:08.541956 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:08.542030 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:08.546202 1582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:08.546285 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:08.546322 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:08.546380 1582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:08.546418 1582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:08.551636 1582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:36591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:38817
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:38817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:08.552802 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:08.553872 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:08.562686 1591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:08.562701 1590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:08.562686 1593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:08.562741 1582 server_base.cc:1034] running on GCE node
I20260806 02:50:08.563575 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:08.564831 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:08.566030 1582 hybrid_clock.cc:648] HybridClock initialized: now 1785984608565999 us; error 45 us; skew 500 ppm
Aug 06 02:50:08 dist-test-slave-gmjp krb5kdc[1561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984608, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:08.568818 1582 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:50:08.569737 1582 webserver.cc:460] Webserver started at http://127.1.183.190:38093/ using document root <none> and password file <none>
I20260806 02:50:08.570320 1582 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:08.570415 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:08.570655 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:08.572839 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "17789740b6fb4b13bf9e356f5c7564ea"
format_stamp: "Formatted at 2026-08-06 02:50:08 on dist-test-slave-gmjp"
server_key: "6f91872016394f2c0e64c6158d7898ef"
server_key_iv: "908c06908d0754944b16d2f04eff962a"
server_key_version: "encryptionkey@0"
I20260806 02:50:08.573453 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "17789740b6fb4b13bf9e356f5c7564ea"
format_stamp: "Formatted at 2026-08-06 02:50:08 on dist-test-slave-gmjp"
server_key: "6f91872016394f2c0e64c6158d7898ef"
server_key_iv: "908c06908d0754944b16d2f04eff962a"
server_key_version: "encryptionkey@0"
I20260806 02:50:08.577181 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:50:08.579672 1604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:08.580960 1582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:50:08.581104 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "17789740b6fb4b13bf9e356f5c7564ea"
format_stamp: "Formatted at 2026-08-06 02:50:08 on dist-test-slave-gmjp"
server_key: "6f91872016394f2c0e64c6158d7898ef"
server_key_iv: "908c06908d0754944b16d2f04eff962a"
server_key_version: "encryptionkey@0"
I20260806 02:50:08.581219 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:08.585810 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:08.588732 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:08.588935 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:08.596056 1582 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:38817
I20260806 02:50:08.596067 1691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:38817 every 8 connection(s)
I20260806 02:50:08.597060 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:08.599871 1692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:08.605372 1692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17789740b6fb4b13bf9e356f5c7564ea: Bootstrap starting.
I20260806 02:50:08.606547 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 1582
I20260806 02:50:08.606717 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:50:08.607014 1758 external_mini_cluster.cc:1428] Setting key 45bbad0a3c136506244eec3fa752b2c5
I20260806 02:50:08.608300 1692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17789740b6fb4b13bf9e356f5c7564ea: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:08.609102 1692 log.cc:824] T 00000000000000000000000000000000 P 17789740b6fb4b13bf9e356f5c7564ea: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:08.611450 1692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17789740b6fb4b13bf9e356f5c7564ea: No bootstrap required, opened a new log
I20260806 02:50:08.612470 1695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:08.608276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59695 (local address 127.1.183.190:38817)
0806 02:50:08.608798 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:08.608814 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:08.608849 (+ 35us) server_negotiation.cc:407] Connection header received
0806 02:50:08.609558 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:08.609578 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:08.609868 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:08.610158 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:08.611079 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:08.611084 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:08.611097 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:08.611108 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:08.611164 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:08.611183 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:08.611281 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:08.611432 (+ 151us) server_negotiation.cc:299] Negotiation successful
0806 02:50:08.611612 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":168,"threads_started":1}
I20260806 02:50:08.615705 1758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:50:08.615803 1692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17789740b6fb4b13bf9e356f5c7564ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17789740b6fb4b13bf9e356f5c7564ea" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38817 } }
I20260806 02:50:08.616088 1692 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17789740b6fb4b13bf9e356f5c7564ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:08.616195 1692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17789740b6fb4b13bf9e356f5c7564ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17789740b6fb4b13bf9e356f5c7564ea, State: Initialized, Role: FOLLOWER
I20260806 02:50:08.616648 1692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17789740b6fb4b13bf9e356f5c7564ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17789740b6fb4b13bf9e356f5c7564ea" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38817 } }
I20260806 02:50:08.616772 1692 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17789740b6fb4b13bf9e356f5c7564ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:08.616860 1692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17789740b6fb4b13bf9e356f5c7564ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:08.616984 1692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17789740b6fb4b13bf9e356f5c7564ea [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:08.618048 1692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17789740b6fb4b13bf9e356f5c7564ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17789740b6fb4b13bf9e356f5c7564ea" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38817 } }
I20260806 02:50:08.618412 1692 leader_election.cc:302] T 00000000000000000000000000000000 P 17789740b6fb4b13bf9e356f5c7564ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17789740b6fb4b13bf9e356f5c7564ea; no voters:
I20260806 02:50:08.618784 1692 leader_election.cc:288] T 00000000000000000000000000000000 P 17789740b6fb4b13bf9e356f5c7564ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:08.618901 1705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17789740b6fb4b13bf9e356f5c7564ea [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:08.619205 1705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17789740b6fb4b13bf9e356f5c7564ea [term 1 LEADER]: Becoming Leader. State: Replica: 17789740b6fb4b13bf9e356f5c7564ea, State: Running, Role: LEADER
I20260806 02:50:08.619609 1705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17789740b6fb4b13bf9e356f5c7564ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17789740b6fb4b13bf9e356f5c7564ea" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38817 } }
I20260806 02:50:08.620290 1695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:08.618118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51519 (local address 127.1.183.190:38817)
0806 02:50:08.618293 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:08.618300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:08.618543 (+ 243us) server_negotiation.cc:407] Connection header received
0806 02:50:08.618706 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:08.618709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:08.618780 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:08.618863 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:08.619694 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:08.619698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:08.619700 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:08.619713 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:08.619742 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:08.619755 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:08.619830 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:08.620113 (+ 283us) server_negotiation.cc:299] Negotiation successful
0806 02:50:08.620175 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 02:50:08.620399 1692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17789740b6fb4b13bf9e356f5c7564ea [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:08.621399 1708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17789740b6fb4b13bf9e356f5c7564ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17789740b6fb4b13bf9e356f5c7564ea. Latest consensus state: current_term: 1 leader_uuid: "17789740b6fb4b13bf9e356f5c7564ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17789740b6fb4b13bf9e356f5c7564ea" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38817 } } }
I20260806 02:50:08.621539 1708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17789740b6fb4b13bf9e356f5c7564ea [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:08.621940 1722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:08.621901 1709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17789740b6fb4b13bf9e356f5c7564ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17789740b6fb4b13bf9e356f5c7564ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17789740b6fb4b13bf9e356f5c7564ea" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 38817 } } }
I20260806 02:50:08.622071 1709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17789740b6fb4b13bf9e356f5c7564ea [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:08.625078 1722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:08.630800 1722 catalog_manager.cc:1347] Generated new cluster ID: 08f75f9844174a4baf327ec0dac03e8f
I20260806 02:50:08.630899 1722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:08.647873 1722 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:08.648253 1722 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:08.655164 1722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17789740b6fb4b13bf9e356f5c7564ea: Generated new TSK 0
I20260806 02:50:08.655841 1722 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:50:08.662731 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 1582
2026-08-06T02:50:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1640 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:50:08 dist-test-slave-gmjp krb5kdc[1740](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:50:08 dist-test-slave-gmjp krb5kdc[1740](info): set up 2 sockets
Aug 06 02:50:08 dist-test-slave-gmjp krb5kdc[1740](info): commencing operation
krb5kdc: starting...
W20260806 02:50:09.892035 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:50:09 dist-test-slave-gmjp krb5kdc[1740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:50:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
I20260806 02:50:10.027405 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:44317
--webserver_interface=127.1.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:34713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:44317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:10.144233 1767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:10.144511 1767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:10.144553 1767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:10.148610 1767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:10.148677 1767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:10.148697 1767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:10.148715 1767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:10.148729 1767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:10.154062 1767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:34713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:44317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:44317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:10.155108 1767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:10.156119 1767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:10.165076 1775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:10.165076 1772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:10.165076 1773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:10.165391 1767 server_base.cc:1034] running on GCE node
I20260806 02:50:10.165984 1767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:10.167121 1767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:10.168285 1767 hybrid_clock.cc:648] HybridClock initialized: now 1785984610168270 us; error 33 us; skew 500 ppm
Aug 06 02:50:10 dist-test-slave-gmjp krb5kdc[1740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984610, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:10.171159 1767 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:50:10.172050 1767 webserver.cc:460] Webserver started at http://127.1.183.190:41283/ using document root <none> and password file <none>
I20260806 02:50:10.172659 1767 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:10.172722 1767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:10.172945 1767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:10.175130 1767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "211f1bcb20c144fba969ca5a43953a09"
format_stamp: "Formatted at 2026-08-06 02:50:10 on dist-test-slave-gmjp"
server_key: "cb166696926258bb5bdd6d95e9b75120"
server_key_iv: "905f793e74e0ed5c44c168ff8dba64b0"
server_key_version: "encryptionkey@0"
I20260806 02:50:10.175732 1767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "211f1bcb20c144fba969ca5a43953a09"
format_stamp: "Formatted at 2026-08-06 02:50:10 on dist-test-slave-gmjp"
server_key: "cb166696926258bb5bdd6d95e9b75120"
server_key_iv: "905f793e74e0ed5c44c168ff8dba64b0"
server_key_version: "encryptionkey@0"
I20260806 02:50:10.179546 1767 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:50:10.182108 1786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:10.183308 1767 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 02:50:10.183459 1767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "211f1bcb20c144fba969ca5a43953a09"
format_stamp: "Formatted at 2026-08-06 02:50:10 on dist-test-slave-gmjp"
server_key: "cb166696926258bb5bdd6d95e9b75120"
server_key_iv: "905f793e74e0ed5c44c168ff8dba64b0"
server_key_version: "encryptionkey@0"
I20260806 02:50:10.183561 1767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:10.207983 1767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:10.211230 1767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:10.211481 1767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:10.218926 1767 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:44317
I20260806 02:50:10.218961 1863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:44317 every 8 connection(s)
I20260806 02:50:10.219898 1767 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:10.222635 1864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:10.225164 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 1767
I20260806 02:50:10.225307 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:50:10.225608 1758 external_mini_cluster.cc:1428] Setting key e13c4cbcb848729171f747bfc39d7b0a
I20260806 02:50:10.228938 1864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 211f1bcb20c144fba969ca5a43953a09: Bootstrap starting.
I20260806 02:50:10.231172 1864 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 211f1bcb20c144fba969ca5a43953a09: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:10.231923 1864 log.cc:824] T 00000000000000000000000000000000 P 211f1bcb20c144fba969ca5a43953a09: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:10.234345 1864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 211f1bcb20c144fba969ca5a43953a09: No bootstrap required, opened a new log
Aug 06 02:50:10 dist-test-slave-gmjp krb5kdc[1740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:50:10.238622 1864 raft_consensus.cc:348] T 00000000000000000000000000000000 P 211f1bcb20c144fba969ca5a43953a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211f1bcb20c144fba969ca5a43953a09" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 44317 } }
I20260806 02:50:10.238946 1864 raft_consensus.cc:374] T 00000000000000000000000000000000 P 211f1bcb20c144fba969ca5a43953a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:10.239012 1864 raft_consensus.cc:729] T 00000000000000000000000000000000 P 211f1bcb20c144fba969ca5a43953a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 211f1bcb20c144fba969ca5a43953a09, State: Initialized, Role: FOLLOWER
I20260806 02:50:10.239521 1864 consensus_queue.cc:260] T 00000000000000000000000000000000 P 211f1bcb20c144fba969ca5a43953a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211f1bcb20c144fba969ca5a43953a09" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 44317 } }
I20260806 02:50:10.239691 1864 raft_consensus.cc:388] T 00000000000000000000000000000000 P 211f1bcb20c144fba969ca5a43953a09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:10.239777 1864 raft_consensus.cc:482] T 00000000000000000000000000000000 P 211f1bcb20c144fba969ca5a43953a09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:10.239899 1864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 211f1bcb20c144fba969ca5a43953a09 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:10.241160 1864 raft_consensus.cc:504] T 00000000000000000000000000000000 P 211f1bcb20c144fba969ca5a43953a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211f1bcb20c144fba969ca5a43953a09" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 44317 } }
I20260806 02:50:10.241526 1864 leader_election.cc:302] T 00000000000000000000000000000000 P 211f1bcb20c144fba969ca5a43953a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 211f1bcb20c144fba969ca5a43953a09; no voters:
I20260806 02:50:10.241714 1869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:10.227054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39529 (local address 127.1.183.190:44317)
0806 02:50:10.227511 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:10.227527 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:10.227564 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:50:10.228272 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:10.228291 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:10.228643 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:10.228981 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:10.229861 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:10.230860 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:10.231812 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:10.232457 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:10.232642 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:10.235448 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:10.235455 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:10.235471 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:10.235502 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:10.238140 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:10.238626 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:10.238634 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:10.238643 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:10.238784 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:10.239084 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:10.239088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:10.239089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:10.239622 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:10.239885 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:10.240282 (+ 397us) server_negotiation.cc:299] Negotiation successful
0806 02:50:10.240610 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":143,"threads_started":1}
I20260806 02:50:10.241891 1864 leader_election.cc:288] T 00000000000000000000000000000000 P 211f1bcb20c144fba969ca5a43953a09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:10.242188 1871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 211f1bcb20c144fba969ca5a43953a09 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:10.242426 1871 raft_consensus.cc:686] T 00000000000000000000000000000000 P 211f1bcb20c144fba969ca5a43953a09 [term 1 LEADER]: Becoming Leader. State: Replica: 211f1bcb20c144fba969ca5a43953a09, State: Running, Role: LEADER
I20260806 02:50:10.242722 1871 consensus_queue.cc:237] T 00000000000000000000000000000000 P 211f1bcb20c144fba969ca5a43953a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211f1bcb20c144fba969ca5a43953a09" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 44317 } }
I20260806 02:50:10.243193 1864 sys_catalog.cc:564] T 00000000000000000000000000000000 P 211f1bcb20c144fba969ca5a43953a09 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:10.244433 1873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 211f1bcb20c144fba969ca5a43953a09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "211f1bcb20c144fba969ca5a43953a09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211f1bcb20c144fba969ca5a43953a09" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 44317 } } }
I20260806 02:50:10.244572 1873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 211f1bcb20c144fba969ca5a43953a09 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:10.244834 1874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 211f1bcb20c144fba969ca5a43953a09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 211f1bcb20c144fba969ca5a43953a09. Latest consensus state: current_term: 1 leader_uuid: "211f1bcb20c144fba969ca5a43953a09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211f1bcb20c144fba969ca5a43953a09" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 44317 } } }
I20260806 02:50:10.244931 1874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 211f1bcb20c144fba969ca5a43953a09 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:10.245254 1883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:10.245296 1758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:50:10.249071 1883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:10.255522 1883 catalog_manager.cc:1347] Generated new cluster ID: c09da0e6358a4284929f30bcff7a8080
I20260806 02:50:10.255597 1883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:10.256610 1869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:10.247467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52343 (local address 127.1.183.190:44317)
0806 02:50:10.247584 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:10.247589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:10.247814 (+ 225us) server_negotiation.cc:407] Connection header received
0806 02:50:10.247971 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:10.247974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:10.248024 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:10.248108 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:10.248997 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:10.249622 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:10.250480 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:10.250725 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:10.250835 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:10.252598 (+ 1763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:10.252602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:10.252605 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:10.252633 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:10.254441 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:10.254933 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:10.254937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:10.254939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:10.255032 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:10.255522 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:10.255526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:10.255527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:10.255842 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:10.255930 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:10.256307 (+ 377us) server_negotiation.cc:299] Negotiation successful
0806 02:50:10.256429 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 02:50:10.272001 1883 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:10.273397 1883 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:10.284328 1883 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 211f1bcb20c144fba969ca5a43953a09: Generated new TSK 0
I20260806 02:50:10.285228 1883 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:50:10.290014 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 1767
2026-08-06T02:50:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1625 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:50:10 dist-test-slave-gmjp krb5kdc[1911](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:50:10 dist-test-slave-gmjp krb5kdc[1911](info): set up 2 sockets
Aug 06 02:50:10 dist-test-slave-gmjp krb5kdc[1911](info): commencing operation
krb5kdc: starting...
W20260806 02:50:11.522923 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.204s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:50:11 dist-test-slave-gmjp krb5kdc[1911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:50:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
I20260806 02:50:11.655026 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:40121
--webserver_interface=127.1.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:38209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:40121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:11.767226 1929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:11.767565 1929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:11.767642 1929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:11.771726 1929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:11.771843 1929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:11.771880 1929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:11.771903 1929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:11.771931 1929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:11.777400 1929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:38209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:40121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:40121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:11.778614 1929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:11.779685 1929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:11.788126 1936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:11.788269 1929 server_base.cc:1034] running on GCE node
W20260806 02:50:11.788189 1935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:11.788203 1938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:11.789009 1929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:11.790071 1929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:11.791239 1929 hybrid_clock.cc:648] HybridClock initialized: now 1785984611791220 us; error 34 us; skew 500 ppm
Aug 06 02:50:11 dist-test-slave-gmjp krb5kdc[1911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984611, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:11.798592 1929 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:50:11.799605 1929 webserver.cc:460] Webserver started at http://127.1.183.190:42463/ using document root <none> and password file <none>
I20260806 02:50:11.800244 1929 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:11.800339 1929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:11.800588 1929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:11.802693 1929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "013278eccc4f4cc09115091fdb51d8a9"
format_stamp: "Formatted at 2026-08-06 02:50:11 on dist-test-slave-gmjp"
server_key: "ea40a5ff7abb38b1521bebed3a7d3b94"
server_key_iv: "0baf573f9c07f752ce4508d7f079c580"
server_key_version: "encryptionkey@0"
I20260806 02:50:11.803323 1929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "013278eccc4f4cc09115091fdb51d8a9"
format_stamp: "Formatted at 2026-08-06 02:50:11 on dist-test-slave-gmjp"
server_key: "ea40a5ff7abb38b1521bebed3a7d3b94"
server_key_iv: "0baf573f9c07f752ce4508d7f079c580"
server_key_version: "encryptionkey@0"
I20260806 02:50:11.807224 1929 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 02:50:11.809844 1951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:11.811131 1929 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:50:11.811280 1929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "013278eccc4f4cc09115091fdb51d8a9"
format_stamp: "Formatted at 2026-08-06 02:50:11 on dist-test-slave-gmjp"
server_key: "ea40a5ff7abb38b1521bebed3a7d3b94"
server_key_iv: "0baf573f9c07f752ce4508d7f079c580"
server_key_version: "encryptionkey@0"
I20260806 02:50:11.811398 1929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:11.842655 1929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:11.845669 1929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:11.845887 1929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:11.853610 1929 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:40121
I20260806 02:50:11.853639 2027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:40121 every 8 connection(s)
I20260806 02:50:11.854604 1929 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:11.857432 2028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:11.861989 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 1929
I20260806 02:50:11.862138 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:50:11.862406 1758 external_mini_cluster.cc:1428] Setting key c06a8fd55091129b7831c1c7105711be
I20260806 02:50:11.863327 2028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 013278eccc4f4cc09115091fdb51d8a9: Bootstrap starting.
I20260806 02:50:11.865863 2028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 013278eccc4f4cc09115091fdb51d8a9: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:11.866633 2028 log.cc:824] T 00000000000000000000000000000000 P 013278eccc4f4cc09115091fdb51d8a9: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:11.868652 2028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 013278eccc4f4cc09115091fdb51d8a9: No bootstrap required, opened a new log
I20260806 02:50:11.871523 2033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:11.863885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37281 (local address 127.1.183.190:40121)
0806 02:50:11.864425 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:11.864441 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:11.864478 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:50:11.865121 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:11.865138 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:11.865498 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:11.865842 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:11.866266 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:11.867307 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:11.868277 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:11.868783 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:11.868978 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:11.869851 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:11.869857 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:11.869871 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:11.870252 (+ 381us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:11.870308 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:11.870330 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:11.870445 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:11.870602 (+ 157us) server_negotiation.cc:299] Negotiation successful
0806 02:50:11.870810 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":152,"threads_started":1}
I20260806 02:50:11.872493 2028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 013278eccc4f4cc09115091fdb51d8a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013278eccc4f4cc09115091fdb51d8a9" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 40121 } }
I20260806 02:50:11.872730 2028 raft_consensus.cc:374] T 00000000000000000000000000000000 P 013278eccc4f4cc09115091fdb51d8a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:11.872816 2028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 013278eccc4f4cc09115091fdb51d8a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 013278eccc4f4cc09115091fdb51d8a9, State: Initialized, Role: FOLLOWER
I20260806 02:50:11.873278 2028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 013278eccc4f4cc09115091fdb51d8a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013278eccc4f4cc09115091fdb51d8a9" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 40121 } }
I20260806 02:50:11.873436 2028 raft_consensus.cc:388] T 00000000000000000000000000000000 P 013278eccc4f4cc09115091fdb51d8a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:11.873507 2028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 013278eccc4f4cc09115091fdb51d8a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:11.873632 2028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 013278eccc4f4cc09115091fdb51d8a9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:11.874492 1758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:50:11.874753 2028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 013278eccc4f4cc09115091fdb51d8a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013278eccc4f4cc09115091fdb51d8a9" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 40121 } }
I20260806 02:50:11.875080 2028 leader_election.cc:302] T 00000000000000000000000000000000 P 013278eccc4f4cc09115091fdb51d8a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 013278eccc4f4cc09115091fdb51d8a9; no voters:
I20260806 02:50:11.875412 2028 leader_election.cc:288] T 00000000000000000000000000000000 P 013278eccc4f4cc09115091fdb51d8a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:11.875476 2035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 013278eccc4f4cc09115091fdb51d8a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:11.875671 2035 raft_consensus.cc:686] T 00000000000000000000000000000000 P 013278eccc4f4cc09115091fdb51d8a9 [term 1 LEADER]: Becoming Leader. State: Replica: 013278eccc4f4cc09115091fdb51d8a9, State: Running, Role: LEADER
I20260806 02:50:11.875976 2035 consensus_queue.cc:237] T 00000000000000000000000000000000 P 013278eccc4f4cc09115091fdb51d8a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013278eccc4f4cc09115091fdb51d8a9" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 40121 } }
I20260806 02:50:11.876732 2028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 013278eccc4f4cc09115091fdb51d8a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:11.877195 2038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 013278eccc4f4cc09115091fdb51d8a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "013278eccc4f4cc09115091fdb51d8a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013278eccc4f4cc09115091fdb51d8a9" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 40121 } } }
I20260806 02:50:11.877323 2038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 013278eccc4f4cc09115091fdb51d8a9 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:11.877682 2037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 013278eccc4f4cc09115091fdb51d8a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 013278eccc4f4cc09115091fdb51d8a9. Latest consensus state: current_term: 1 leader_uuid: "013278eccc4f4cc09115091fdb51d8a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013278eccc4f4cc09115091fdb51d8a9" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 40121 } } }
I20260806 02:50:11.877815 2037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 013278eccc4f4cc09115091fdb51d8a9 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:11.879180 2047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:11.882592 2047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:11.883888 2033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:11.876865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56633 (local address 127.1.183.190:40121)
0806 02:50:11.876987 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:11.876993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:11.879145 (+ 2152us) server_negotiation.cc:407] Connection header received
0806 02:50:11.879253 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:11.879256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:11.879301 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:11.879381 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:11.879913 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:11.880696 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:11.881829 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:11.882160 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:11.882257 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:11.883174 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:11.883178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:11.883181 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:11.883194 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:11.883220 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:11.883233 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:11.883295 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:11.883701 (+ 406us) server_negotiation.cc:299] Negotiation successful
0806 02:50:11.883752 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 02:50:11.887909 2047 catalog_manager.cc:1347] Generated new cluster ID: 46226ce3047a4862b40177c67b5fef17
I20260806 02:50:11.887969 2047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:11.895927 2047 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:11.896835 2047 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:11.905167 2047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 013278eccc4f4cc09115091fdb51d8a9: Generated new TSK 0
I20260806 02:50:11.905876 2047 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:50:11.911437 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 1929
2026-08-06T02:50:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1620 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:50:11 dist-test-slave-gmjp krb5kdc[2072](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:50:11 dist-test-slave-gmjp krb5kdc[2072](info): set up 2 sockets
Aug 06 02:50:11 dist-test-slave-gmjp krb5kdc[2072](info): commencing operation
krb5kdc: starting...
W20260806 02:50:13.158347 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.219s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:50:13 dist-test-slave-gmjp krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:50:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
I20260806 02:50:13.293463 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:35657
--webserver_interface=127.1.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:38517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:35657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:13.409996 2093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:13.410393 2093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:13.410487 2093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:13.414865 2093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:13.414990 2093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:13.415028 2093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:13.415071 2093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:13.415103 2093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:13.420765 2093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:38517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:35657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:35657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:13.421980 2093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:13.423122 2093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:13.431946 2104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:13.432017 2101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:13.432129 2093 server_base.cc:1034] running on GCE node
W20260806 02:50:13.432384 2100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:13.432989 2093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:13.434120 2093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:13.435328 2093 hybrid_clock.cc:648] HybridClock initialized: now 1785984613435308 us; error 32 us; skew 500 ppm
Aug 06 02:50:13 dist-test-slave-gmjp krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984613, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:13.438308 2093 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:50:13.439306 2093 webserver.cc:460] Webserver started at http://127.1.183.190:33945/ using document root <none> and password file <none>
I20260806 02:50:13.439893 2093 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:13.439985 2093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:13.440295 2093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:13.442647 2093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "66417b63cb814bfe9ead6ec69e58f2f5"
format_stamp: "Formatted at 2026-08-06 02:50:13 on dist-test-slave-gmjp"
server_key: "346b5acd5126055f13f845dcd6c064ec"
server_key_iv: "f83a3dc4ac8e1815175a487c3c8e17d1"
server_key_version: "encryptionkey@0"
I20260806 02:50:13.443270 2093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "66417b63cb814bfe9ead6ec69e58f2f5"
format_stamp: "Formatted at 2026-08-06 02:50:13 on dist-test-slave-gmjp"
server_key: "346b5acd5126055f13f845dcd6c064ec"
server_key_iv: "f83a3dc4ac8e1815175a487c3c8e17d1"
server_key_version: "encryptionkey@0"
I20260806 02:50:13.447259 2093 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 02:50:13.449934 2116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:13.451270 2093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260806 02:50:13.451424 2093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "66417b63cb814bfe9ead6ec69e58f2f5"
format_stamp: "Formatted at 2026-08-06 02:50:13 on dist-test-slave-gmjp"
server_key: "346b5acd5126055f13f845dcd6c064ec"
server_key_iv: "f83a3dc4ac8e1815175a487c3c8e17d1"
server_key_version: "encryptionkey@0"
I20260806 02:50:13.451550 2093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:13.456380 2093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:13.459700 2093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:13.459930 2093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:13.467391 2093 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:35657
I20260806 02:50:13.467468 2190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:35657 every 8 connection(s)
I20260806 02:50:13.468413 2093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:13.471057 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 2093
I20260806 02:50:13.471220 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:50:13.471429 2191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:13.471562 1758 external_mini_cluster.cc:1428] Setting key 1e4170e77b0c2f7539d26ff6fcea4ec6
I20260806 02:50:13.477150 2195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:13.473073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60393 (local address 127.1.183.190:35657)
0806 02:50:13.473519 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:13.473535 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:13.473571 (+ 36us) server_negotiation.cc:407] Connection header received
0806 02:50:13.474204 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:13.474220 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:13.474500 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:13.474792 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:13.475775 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:13.475780 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:13.475793 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:13.475805 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:13.475864 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:13.475883 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:13.476007 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:13.476122 (+ 115us) server_negotiation.cc:299] Negotiation successful
0806 02:50:13.476376 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":153,"threads_started":1}
I20260806 02:50:13.478555 2191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66417b63cb814bfe9ead6ec69e58f2f5: Bootstrap starting.
I20260806 02:50:13.480053 1758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:50:13.481092 2191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66417b63cb814bfe9ead6ec69e58f2f5: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:13.482025 2191 log.cc:824] T 00000000000000000000000000000000 P 66417b63cb814bfe9ead6ec69e58f2f5: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:13.484120 2195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:13.482241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60637 (local address 127.1.183.190:35657)
0806 02:50:13.482397 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:13.482401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:13.482569 (+ 168us) server_negotiation.cc:407] Connection header received
0806 02:50:13.482713 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:13.482716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:13.482790 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:13.482879 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:13.483673 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:13.483677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:13.483680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:50:13.483692 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:13.483725 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:50:13.483736 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:13.483866 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:13.483950 (+ 84us) server_negotiation.cc:299] Negotiation successful
0806 02:50:13.483999 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:50:13.484557 2191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66417b63cb814bfe9ead6ec69e58f2f5: No bootstrap required, opened a new log
I20260806 02:50:13.488297 2191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66417b63cb814bfe9ead6ec69e58f2f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66417b63cb814bfe9ead6ec69e58f2f5" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35657 } }
I20260806 02:50:13.488545 2191 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66417b63cb814bfe9ead6ec69e58f2f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:50:13.488624 2191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66417b63cb814bfe9ead6ec69e58f2f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66417b63cb814bfe9ead6ec69e58f2f5, State: Initialized, Role: FOLLOWER
I20260806 02:50:13.489149 2191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66417b63cb814bfe9ead6ec69e58f2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66417b63cb814bfe9ead6ec69e58f2f5" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35657 } }
I20260806 02:50:13.489341 2191 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66417b63cb814bfe9ead6ec69e58f2f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:50:13.489421 2191 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66417b63cb814bfe9ead6ec69e58f2f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:50:13.489526 2191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66417b63cb814bfe9ead6ec69e58f2f5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:13.490522 2191 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66417b63cb814bfe9ead6ec69e58f2f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66417b63cb814bfe9ead6ec69e58f2f5" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35657 } }
I20260806 02:50:13.490888 2191 leader_election.cc:302] T 00000000000000000000000000000000 P 66417b63cb814bfe9ead6ec69e58f2f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66417b63cb814bfe9ead6ec69e58f2f5; no voters:
I20260806 02:50:13.491297 2191 leader_election.cc:288] T 00000000000000000000000000000000 P 66417b63cb814bfe9ead6ec69e58f2f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:50:13.491480 2207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66417b63cb814bfe9ead6ec69e58f2f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:13.491743 2207 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66417b63cb814bfe9ead6ec69e58f2f5 [term 1 LEADER]: Becoming Leader. State: Replica: 66417b63cb814bfe9ead6ec69e58f2f5, State: Running, Role: LEADER
I20260806 02:50:13.492071 2207 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66417b63cb814bfe9ead6ec69e58f2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66417b63cb814bfe9ead6ec69e58f2f5" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35657 } }
I20260806 02:50:13.492949 2191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66417b63cb814bfe9ead6ec69e58f2f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:13.495139 2208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66417b63cb814bfe9ead6ec69e58f2f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66417b63cb814bfe9ead6ec69e58f2f5. Latest consensus state: current_term: 1 leader_uuid: "66417b63cb814bfe9ead6ec69e58f2f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66417b63cb814bfe9ead6ec69e58f2f5" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35657 } } }
I20260806 02:50:13.495281 2208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66417b63cb814bfe9ead6ec69e58f2f5 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:13.495317 2210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66417b63cb814bfe9ead6ec69e58f2f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66417b63cb814bfe9ead6ec69e58f2f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66417b63cb814bfe9ead6ec69e58f2f5" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35657 } } }
I20260806 02:50:13.495421 2210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66417b63cb814bfe9ead6ec69e58f2f5 [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:13.495767 2226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:13.498485 2226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:13.503594 2226 catalog_manager.cc:1347] Generated new cluster ID: add28dc5d48348c8b593b7f3a34889a9
I20260806 02:50:13.503664 2226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:13.522408 2226 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:13.522794 2226 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:13.529533 2226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66417b63cb814bfe9ead6ec69e58f2f5: Generated new TSK 0
I20260806 02:50:13.530246 2226 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:50:13.536746 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 2093
2026-08-06T02:50:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1620 ms)
[----------] 18 tests from AuthTokenIssuingTest (30313 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:50:13 dist-test-slave-gmjp krb5kdc[2237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:50:13 dist-test-slave-gmjp krb5kdc[2237](info): set up 2 sockets
Aug 06 02:50:13 dist-test-slave-gmjp krb5kdc[2237](info): commencing operation
krb5kdc: starting...
W20260806 02:50:14.785403 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.226s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:50:14 dist-test-slave-gmjp krb5kdc[2237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:50:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
I20260806 02:50:14.916034 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:35499
--webserver_interface=127.1.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:33093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:35499,127.1.183.189:43615,127.1.183.188:35253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:15.031282 2255 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:50:15.031667 2255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:15.031749 2255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:15.031827 2255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:15.035862 2255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:15.035972 2255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:15.036011 2255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:15.036049 2255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:15.036084 2255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:15.041479 2255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:33093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:35499,127.1.183.189:43615,127.1.183.188:35253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:35499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:15.042672 2255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:15.043745 2255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:15.052462 2269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:15.052522 2268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:15.052569 2255 server_base.cc:1034] running on GCE node
W20260806 02:50:15.052531 2272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:15.053406 2255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:15.054538 2255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:15.055725 2255 hybrid_clock.cc:648] HybridClock initialized: now 1785984615055698 us; error 40 us; skew 500 ppm
Aug 06 02:50:15 dist-test-slave-gmjp krb5kdc[2237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984615, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:15.058561 2255 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:50:15.059489 2255 webserver.cc:460] Webserver started at http://127.1.183.190:41515/ using document root <none> and password file <none>
I20260806 02:50:15.060083 2255 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:15.060191 2255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:15.060400 2255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:15.062552 2255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "c4ff79998ad0412bb554589b7b9d385e"
format_stamp: "Formatted at 2026-08-06 02:50:15 on dist-test-slave-gmjp"
server_key: "1fd129615a1bab55723cb64f9f6640f1"
server_key_iv: "35946dbd8aa36400e116e26ba83d3c0b"
server_key_version: "encryptionkey@0"
I20260806 02:50:15.063097 2255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "c4ff79998ad0412bb554589b7b9d385e"
format_stamp: "Formatted at 2026-08-06 02:50:15 on dist-test-slave-gmjp"
server_key: "1fd129615a1bab55723cb64f9f6640f1"
server_key_iv: "35946dbd8aa36400e116e26ba83d3c0b"
server_key_version: "encryptionkey@0"
I20260806 02:50:15.066807 2255 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260806 02:50:15.069219 2280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:15.070472 2255 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:50:15.070605 2255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "c4ff79998ad0412bb554589b7b9d385e"
format_stamp: "Formatted at 2026-08-06 02:50:15 on dist-test-slave-gmjp"
server_key: "1fd129615a1bab55723cb64f9f6640f1"
server_key_iv: "35946dbd8aa36400e116e26ba83d3c0b"
server_key_version: "encryptionkey@0"
I20260806 02:50:15.070709 2255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:15.095122 2255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:15.098289 2255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:15.098536 2255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:15.106204 2255 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:35499
I20260806 02:50:15.106536 2357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:35499 every 8 connection(s)
I20260806 02:50:15.107262 2255 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:15.110136 2358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:15.113708 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 2255
I20260806 02:50:15.113847 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:50:15.114116 1758 external_mini_cluster.cc:1428] Setting key 35fb034b7031817f58169c65b54c6adb
I20260806 02:50:15.114125 2358 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35499 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.183.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.189@KRBTEST.COM" created.
Aug 06 02:50:15 dist-test-slave-gmjp krb5kdc[2237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984615, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:50:15.132900 2365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.116571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42964 (local address 127.1.183.190:35499)
0806 02:50:15.117149 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:15.117162 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:15.117210 (+ 48us) server_negotiation.cc:407] Connection header received
0806 02:50:15.118275 (+ 1065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:15.118283 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:15.118491 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:15.118695 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:15.119558 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.120598 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.121589 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.121958 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.122130 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:15.127617 (+ 5487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:15.127624 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:15.127638 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:15.127664 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:15.129568 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:15.130080 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:15.130089 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:15.130094 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:15.130247 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:15.130607 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:15.130613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:15.130618 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:15.131221 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:15.131373 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:15.131773 (+ 400us) server_negotiation.cc:299] Negotiation successful
0806 02:50:15.132088 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":457,"spinlock_wait_cycles":14080,"thread_start_us":159,"threads_started":1}
I20260806 02:50:15.132908 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.116570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35499 (local address 127.0.0.1:42964)
0806 02:50:15.117059 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:15.117112 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:15.117968 (+ 856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:15.118693 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:15.118698 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:15.119148 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:15.119402 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:15.119415 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.120731 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.120739 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:15.121463 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:15.121472 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.122118 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.122121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:15.122229 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:15.122845 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:15.122873 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:15.127391 (+ 4518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:15.129695 (+ 2304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:15.129707 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:15.129719 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:15.129957 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:15.130371 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:15.130375 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:15.130377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:15.130504 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:15.131347 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:15.131353 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:15.131490 (+ 137us) client_negotiation.cc:769] Sending connection context
0806 02:50:15.131637 (+ 147us) client_negotiation.cc:240] Negotiation successful
0806 02:50:15.131933 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":146,"threads_started":1}
I20260806 02:50:15.135710 2358 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.183.189" port: 43615 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:50:15.136551 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.136297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:43615 (local address 127.0.0.1:46016)
0806 02:50:15.136424 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:15.136494 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.189:43615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 02:50:15.136742 2284 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.183.189:43615: connect: Connection refused (error 111)
I20260806 02:50:15.137557 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.137349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:43615 (local address 127.0.0.1:46030)
0806 02:50:15.137459 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:15.137506 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.189:43615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 02:50:15.137724 2358 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.183.189:43615: Network error: Client connection negotiation failed: client connection to 127.1.183.189:43615: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.189.keytab.
Entry for principal kudu/127.1.183.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.189.keytab.
I20260806 02:50:15.146685 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.189:43615
--webserver_interface=127.1.183.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.189.keytab
--principal=kudu/127.1.183.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:33093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:35499,127.1.183.189:43615,127.1.183.188:35253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:50:15.186987 2358 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.183.189" port: 43615 } attempt: 1
I20260806 02:50:15.188061 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.187660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:43615 (local address 127.0.0.1:46042)
0806 02:50:15.187920 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:15.187981 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.189:43615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260806 02:50:15.188879 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.188655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:43615 (local address 127.0.0.1:46058)
0806 02:50:15.188765 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:15.188813 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.189:43615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 02:50:15.189033 2358 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.183.189:43615: Network error: Client connection negotiation failed: client connection to 127.1.183.189:43615: connect: Connection refused (error 111)
I20260806 02:50:15.257202 2358 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.183.189" port: 43615 } attempt: 2
I20260806 02:50:15.258165 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.257779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:43615 (local address 127.0.0.1:46070)
0806 02:50:15.257992 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:15.258070 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.189:43615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 02:50:15.258980 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.258720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:43615 (local address 127.0.0.1:46076)
0806 02:50:15.258839 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:15.258895 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.189:43615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 02:50:15.259132 2358 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.183.189:43615: Network error: Client connection negotiation failed: client connection to 127.1.183.189:43615: connect: Connection refused (error 111)
W20260806 02:50:15.265036 2370 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:50:15.265321 2370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:15.265373 2370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:15.265501 2370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:15.269615 2370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:15.269666 2370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:15.269686 2370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:15.269735 2370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:15.269765 2370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:15.275131 2370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:33093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:35499,127.1.183.189:43615,127.1.183.188:35253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.189.keytab
--principal=kudu/127.1.183.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.189:43615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.183.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:15.276330 2370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:15.277367 2370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:15.286181 2376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:15.286186 2377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:15.286217 2379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:15.286465 2370 server_base.cc:1034] running on GCE node
I20260806 02:50:15.287113 2370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:15.288393 2370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:15.289561 2370 hybrid_clock.cc:648] HybridClock initialized: now 1785984615289548 us; error 32 us; skew 500 ppm
Aug 06 02:50:15 dist-test-slave-gmjp krb5kdc[2237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984615, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:15.292515 2370 init.cc:377] Logged in from keytab as kudu/127.1.183.189@KRBTEST.COM (short username kudu)
I20260806 02:50:15.293397 2370 webserver.cc:460] Webserver started at http://127.1.183.189:42829/ using document root <none> and password file <none>
I20260806 02:50:15.293958 2370 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:15.294020 2370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:15.294198 2370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:15.296366 2370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-1/data/instance:
uuid: "b6d28389e22246dc8c4374504701afa8"
format_stamp: "Formatted at 2026-08-06 02:50:15 on dist-test-slave-gmjp"
server_key: "f085b6b0b4e5f231231285456d4998d7"
server_key_iv: "a0f5646ca0ce10fc8fab50244afee485"
server_key_version: "encryptionkey@0"
I20260806 02:50:15.296923 2370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-1/wal/instance:
uuid: "b6d28389e22246dc8c4374504701afa8"
format_stamp: "Formatted at 2026-08-06 02:50:15 on dist-test-slave-gmjp"
server_key: "f085b6b0b4e5f231231285456d4998d7"
server_key_iv: "a0f5646ca0ce10fc8fab50244afee485"
server_key_version: "encryptionkey@0"
I20260806 02:50:15.300765 2370 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:50:15.303158 2387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:15.304507 2370 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:50:15.304642 2370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-1/wal
uuid: "b6d28389e22246dc8c4374504701afa8"
format_stamp: "Formatted at 2026-08-06 02:50:15 on dist-test-slave-gmjp"
server_key: "f085b6b0b4e5f231231285456d4998d7"
server_key_iv: "a0f5646ca0ce10fc8fab50244afee485"
server_key_version: "encryptionkey@0"
I20260806 02:50:15.304777 2370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:15.319211 2370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:15.322808 2370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:15.323017 2370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:15.330615 2370 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.189:43615
I20260806 02:50:15.330914 2463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.189:43615 every 8 connection(s)
I20260806 02:50:15.331697 2370 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-1/data/info.pb
I20260806 02:50:15.333627 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 2370
I20260806 02:50:15.333768 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-1/wal/instance
I20260806 02:50:15.334074 1758 external_mini_cluster.cc:1428] Setting key daaf9c9a9ecfd81b0938af6f4763b2fd
I20260806 02:50:15.335088 2464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:15.339358 2464 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35499 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.183.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.188@KRBTEST.COM" created.
Aug 06 02:50:15 dist-test-slave-gmjp krb5kdc[2237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984615, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.189@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:50:15.350327 2358 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.183.189" port: 43615 } attempt: 3
I20260806 02:50:15.355206 2365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.341413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42966 (local address 127.1.183.190:35499)
0806 02:50:15.341590 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:15.341595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:15.342598 (+ 1003us) server_negotiation.cc:407] Connection header received
0806 02:50:15.343752 (+ 1154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:15.343755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:15.343805 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:15.343891 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:15.345238 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.345863 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.347089 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.347525 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.347655 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:15.350681 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:15.350685 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:15.350687 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:15.350712 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:15.352609 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:15.353286 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:15.353290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:15.353291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:15.353380 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:15.353852 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:15.353856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:15.353858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:15.354068 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:15.354169 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:15.354876 (+ 707us) server_negotiation.cc:299] Negotiation successful
0806 02:50:15.355019 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 02:50:15.356369 2468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.341814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35499 (local address 127.0.0.1:42966)
0806 02:50:15.342396 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:15.342463 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:15.343555 (+ 1092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:15.344135 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:15.344141 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:15.344538 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:15.345096 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:15.345111 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.346014 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.346021 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:15.346955 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:15.346965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.347683 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.347686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:15.347943 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:15.348642 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:15.348670 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:15.350452 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:15.352740 (+ 2288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:15.352747 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:15.352764 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:15.353168 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:15.353491 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:15.353497 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:15.353501 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:15.353692 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:15.354199 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:15.354208 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:15.354554 (+ 346us) client_negotiation.cc:769] Sending connection context
0806 02:50:15.354805 (+ 251us) client_negotiation.cc:240] Negotiation successful
0806 02:50:15.355212 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":119,"threads_started":1}
Aug 06 02:50:15 dist-test-slave-gmjp krb5kdc[2237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984615, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for kudu/127.1.183.189@KRBTEST.COM
I20260806 02:50:15.358765 2464 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.183.189" port: 43615 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.188.keytab.
Entry for principal kudu/127.1.183.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.188.keytab.
I20260806 02:50:15.363125 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.350821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:43615 (local address 127.0.0.1:46090)
0806 02:50:15.351062 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:15.351078 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:15.351163 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:15.352204 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:15.352211 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:15.352506 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:15.352645 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:15.352655 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.353769 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.353772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:15.354462 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:15.354470 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.355242 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.355249 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:15.355388 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:15.355842 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:15.355855 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:15.358945 (+ 3090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:15.361195 (+ 2250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:15.361199 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:15.361201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:15.361516 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:15.361825 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:15.361827 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:15.361829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:15.361905 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:15.362457 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:15.362460 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:15.362596 (+ 136us) client_negotiation.cc:769] Sending connection context
0806 02:50:15.362715 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 02:50:15.362885 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142}
I20260806 02:50:15.363754 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.188:35253
--webserver_interface=127.1.183.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.188.keytab
--principal=kudu/127.1.183.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:33093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:35499,127.1.183.189:43615,127.1.183.188:35253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:50:15.363817 2474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.350928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46090 (local address 127.1.183.189:43615)
0806 02:50:15.351652 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:15.351666 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:15.351695 (+ 29us) server_negotiation.cc:407] Connection header received
0806 02:50:15.351747 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:15.351754 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:15.352008 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:15.352203 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:15.352774 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.353648 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.354575 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.355100 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.355768 (+ 668us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:15.359079 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:15.359089 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:15.359102 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:15.359134 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:15.361095 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:15.361624 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:15.361630 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:15.361632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:15.361723 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:15.362011 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:15.362015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:15.362017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:15.362280 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:15.362439 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:15.363416 (+ 977us) server_negotiation.cc:299] Negotiation successful
0806 02:50:15.363584 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":461,"threads_started":1}
I20260806 02:50:15.365944 2358 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35253 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:50:15 dist-test-slave-gmjp krb5kdc[2237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984615, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.189@KRBTEST.COM for kudu/127.1.183.189@KRBTEST.COM
I20260806 02:50:15.366840 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.366516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:35253 (local address 127.0.0.1:50946)
0806 02:50:15.366708 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:15.366763 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:35253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260806 02:50:15.368016 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.367400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:35253 (local address 127.0.0.1:50956)
0806 02:50:15.367872 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:15.367935 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:35253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381}
W20260806 02:50:15.368367 2358 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.183.188:35253: Network error: Client connection negotiation failed: client connection to 127.1.183.188:35253: connect: Connection refused (error 111)
I20260806 02:50:15.370747 2468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.359277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:43615 (local address 127.0.0.1:46102)
0806 02:50:15.359720 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:15.359745 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:15.359822 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:15.360073 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:15.360076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:15.360280 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:15.360463 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:15.360475 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.361384 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.361386 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:15.362241 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:15.362251 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.362912 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.362915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:15.363052 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:15.363660 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:15.363679 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:15.366990 (+ 3311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:15.369267 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:15.369270 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:15.369271 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:15.369469 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:15.369804 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:15.369807 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:15.369808 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:15.369879 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:15.370361 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:15.370362 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:15.370449 (+ 87us) client_negotiation.cc:769] Sending connection context
0806 02:50:15.370527 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 02:50:15.370632 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":19}
I20260806 02:50:15.371052 2475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.359501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46102 (local address 127.1.183.189:43615)
0806 02:50:15.359800 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:15.359809 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:15.359822 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:50:15.359898 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:15.359903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:15.359939 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:15.360025 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:15.360620 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.361275 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.362355 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.362813 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.362915 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:15.367163 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:15.367166 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:15.367168 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:15.367188 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:15.369162 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:15.369574 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:15.369579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:15.369582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:15.369695 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:15.369973 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:15.369978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:15.369983 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:15.370260 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:15.370376 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:15.370621 (+ 245us) server_negotiation.cc:299] Negotiation successful
0806 02:50:15.370779 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":230,"thread_start_us":101,"threads_started":1}
I20260806 02:50:15.371773 2464 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35253 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:50:15.372522 2468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.372268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:35253 (local address 127.0.0.1:50972)
0806 02:50:15.372378 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:15.372470 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:35253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 02:50:15.372797 2388 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.183.188:35253: connect: Connection refused (error 111)
I20260806 02:50:15.373759 2468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.373561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:35253 (local address 127.0.0.1:50984)
0806 02:50:15.373668 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:15.373711 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:35253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 02:50:15.374074 2464 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.183.188:35253: Network error: Client connection negotiation failed: client connection to 127.1.183.188:35253: connect: Connection refused (error 111)
I20260806 02:50:15.399616 2358 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35253 } attempt: 1
I20260806 02:50:15.400657 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.400289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:35253 (local address 127.0.0.1:50992)
0806 02:50:15.400476 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:15.400556 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:35253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 02:50:15.401461 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.401228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:35253 (local address 127.0.0.1:51006)
0806 02:50:15.401338 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:15.401395 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:35253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 02:50:15.401608 2358 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.183.188:35253: Network error: Client connection negotiation failed: client connection to 127.1.183.188:35253: connect: Connection refused (error 111)
I20260806 02:50:15.423408 2464 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35253 } attempt: 1
I20260806 02:50:15.424393 2468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.424042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:35253 (local address 127.0.0.1:51020)
0806 02:50:15.424234 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:15.424309 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:35253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 02:50:15.425218 2468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.424992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:35253 (local address 127.0.0.1:51036)
0806 02:50:15.425108 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:15.425164 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:35253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 02:50:15.425402 2464 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.183.188:35253: Network error: Client connection negotiation failed: client connection to 127.1.183.188:35253: connect: Connection refused (error 111)
I20260806 02:50:15.476774 2358 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35253 } attempt: 2
I20260806 02:50:15.477700 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.477340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:35253 (local address 127.0.0.1:51038)
0806 02:50:15.477527 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:15.477603 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:35253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260806 02:50:15.478474 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.478224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:35253 (local address 127.0.0.1:51044)
0806 02:50:15.478344 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:15.478398 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:35253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 02:50:15.478637 2358 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.183.188:35253: Network error: Client connection negotiation failed: client connection to 127.1.183.188:35253: connect: Connection refused (error 111)
W20260806 02:50:15.485453 2476 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:50:15.485751 2476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:15.485774 2476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:15.485813 2476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:15.489858 2476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:15.489917 2476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:15.489936 2476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:15.489953 2476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:15.489969 2476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:15.493566 2464 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35253 } attempt: 2
I20260806 02:50:15.494477 2468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.494154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:35253 (local address 127.0.0.1:51048)
0806 02:50:15.494323 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:15.494389 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:35253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260806 02:50:15.495311 2468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.495109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:35253 (local address 127.0.0.1:51058)
0806 02:50:15.495213 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:15.495258 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:35253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 02:50:15.495452 2464 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.183.188:35253: Network error: Client connection negotiation failed: client connection to 127.1.183.188:35253: connect: Connection refused (error 111)
I20260806 02:50:15.496194 2476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:33093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:35499,127.1.183.189:43615,127.1.183.188:35253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.188.keytab
--principal=kudu/127.1.183.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.188:35253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.183.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:15.497314 2476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:15.498322 2476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:15.508210 2486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:15.508213 2484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:15.508214 2488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:15.508291 2476 server_base.cc:1034] running on GCE node
I20260806 02:50:15.509171 2476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:15.510306 2476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:15.511520 2476 hybrid_clock.cc:648] HybridClock initialized: now 1785984615511503 us; error 40 us; skew 500 ppm
Aug 06 02:50:15 dist-test-slave-gmjp krb5kdc[2237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984615, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:15.514519 2476 init.cc:377] Logged in from keytab as kudu/127.1.183.188@KRBTEST.COM (short username kudu)
I20260806 02:50:15.515412 2476 webserver.cc:460] Webserver started at http://127.1.183.188:42369/ using document root <none> and password file <none>
I20260806 02:50:15.515993 2476 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:15.516057 2476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:15.516305 2476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:15.518467 2476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-2/data/instance:
uuid: "f3fbe2801b9d4d219600e08936c8ad79"
format_stamp: "Formatted at 2026-08-06 02:50:15 on dist-test-slave-gmjp"
server_key: "89061480ba0afa0141666b7c7b4e26d8"
server_key_iv: "7b48cd8146333a7f680cb9ed0077dfc6"
server_key_version: "encryptionkey@0"
I20260806 02:50:15.519068 2476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-2/wal/instance:
uuid: "f3fbe2801b9d4d219600e08936c8ad79"
format_stamp: "Formatted at 2026-08-06 02:50:15 on dist-test-slave-gmjp"
server_key: "89061480ba0afa0141666b7c7b4e26d8"
server_key_iv: "7b48cd8146333a7f680cb9ed0077dfc6"
server_key_version: "encryptionkey@0"
I20260806 02:50:15.522894 2476 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:50:15.525382 2496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:15.526724 2476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 02:50:15.526872 2476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-2/wal
uuid: "f3fbe2801b9d4d219600e08936c8ad79"
format_stamp: "Formatted at 2026-08-06 02:50:15 on dist-test-slave-gmjp"
server_key: "89061480ba0afa0141666b7c7b4e26d8"
server_key_iv: "7b48cd8146333a7f680cb9ed0077dfc6"
server_key_version: "encryptionkey@0"
I20260806 02:50:15.526989 2476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:15.542598 2476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:15.546025 2476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:15.546247 2476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:15.553736 2476 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.188:35253
I20260806 02:50:15.554054 2566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.188:35253 every 8 connection(s)
I20260806 02:50:15.554729 2476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-2/data/info.pb
I20260806 02:50:15.557554 2568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:15.561334 2568 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35499 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:50:15.562525 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 2476
I20260806 02:50:15.562680 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984313887056-1758-0/minicluster-data/master-2/wal/instance
I20260806 02:50:15.562934 1758 external_mini_cluster.cc:1428] Setting key a32c3eaa9020d02b6b4c415651640cf2
Aug 06 02:50:15 dist-test-slave-gmjp krb5kdc[2237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984615, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.188@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
Aug 06 02:50:15 dist-test-slave-gmjp krb5kdc[2237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.188@KRBTEST.COM
I20260806 02:50:15.575589 2365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.563406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42978 (local address 127.1.183.190:35499)
0806 02:50:15.563581 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:15.563585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:15.564625 (+ 1040us) server_negotiation.cc:407] Connection header received
0806 02:50:15.565406 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:15.565408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:15.565454 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:15.565524 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:15.566482 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.567009 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.568147 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.568527 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.568644 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:15.571350 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:15.571353 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:15.571356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:15.571382 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:15.573204 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:15.573862 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:15.573866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:15.573875 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:15.573964 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:15.574316 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:15.574319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:15.574321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:15.574658 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:15.574760 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:15.575289 (+ 529us) server_negotiation.cc:299] Negotiation successful
0806 02:50:15.575414 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:50:15.576548 2575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.563799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51060 (local address 127.1.183.188:35253)
0806 02:50:15.564305 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:15.564325 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:15.564365 (+ 40us) server_negotiation.cc:407] Connection header received
0806 02:50:15.565190 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:15.565213 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:15.565449 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:15.565646 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:15.566405 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.567430 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.568503 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.568970 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.569088 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:15.571857 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:15.571863 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:15.571893 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:15.571921 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:15.573800 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:15.574266 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:15.574273 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:15.574274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:15.574358 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:15.574671 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:15.574676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:15.574677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:15.575066 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:15.575297 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:15.575517 (+ 220us) server_negotiation.cc:299] Negotiation successful
0806 02:50:15.575660 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":354,"spinlock_wait_cycles":53376,"thread_start_us":178,"threads_started":1}
I20260806 02:50:15.576551 2574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.563816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:35499 (local address 127.0.0.1:42978)
0806 02:50:15.564471 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:15.564517 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:15.565190 (+ 673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:15.565602 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:15.565608 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:15.566061 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:15.566313 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:15.566349 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.567124 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.567128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:15.568019 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:15.568028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.568703 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.568708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:15.568889 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:15.569509 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:15.569535 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:15.571179 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:15.573316 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:15.573325 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:15.573334 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:15.573756 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:15.574063 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:15.574067 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:15.574069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:15.574212 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:15.574759 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:15.574766 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:15.575064 (+ 298us) client_negotiation.cc:769] Sending connection context
0806 02:50:15.575242 (+ 178us) client_negotiation.cc:240] Negotiation successful
0806 02:50:15.575571 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":2,"spinlock_wait_cycles":1792,"thread_start_us":132,"threads_started":1}
I20260806 02:50:15.577795 2358 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35253 } attempt: 3
I20260806 02:50:15.577952 2568 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.183.189" port: 43615 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:50:15 dist-test-slave-gmjp krb5kdc[2237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984615, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for kudu/127.1.183.188@KRBTEST.COM
Aug 06 02:50:15 dist-test-slave-gmjp krb5kdc[2237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984615, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.188@KRBTEST.COM for kudu/127.1.183.189@KRBTEST.COM
Aug 06 02:50:15 dist-test-slave-gmjp krb5kdc[2237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:50:15.586614 2464 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35253 } attempt: 3
I20260806 02:50:15.588327 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.578435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:35253 (local address 127.0.0.1:51066)
0806 02:50:15.578775 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:15.578791 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:15.578889 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:15.579181 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:15.579187 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:15.579384 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:15.579498 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:15.579504 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.580595 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.580601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:15.581444 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:15.581455 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.581999 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.582002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:15.582148 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:15.582644 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:15.582663 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:15.584505 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:15.586538 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:15.586543 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:15.586545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:15.586784 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:15.587122 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:15.587125 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:15.587127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:15.587220 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:15.587738 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:15.587743 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:15.587855 (+ 112us) client_negotiation.cc:769] Sending connection context
0806 02:50:15.587946 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 02:50:15.588086 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":47,"spinlock_wait_cycles":1280}
I20260806 02:50:15.588932 2575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.578616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51066 (local address 127.1.183.188:35253)
0806 02:50:15.578722 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:15.578727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:15.578974 (+ 247us) server_negotiation.cc:407] Connection header received
0806 02:50:15.579014 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:15.579017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:15.579060 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:15.579139 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:15.579650 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.580482 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.581578 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.581880 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.581998 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:15.584643 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:15.584646 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:15.584648 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:15.584668 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:15.586393 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:15.586919 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:15.586924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:15.586926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:15.587013 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:15.587341 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:15.587347 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:15.587352 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:15.587615 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:15.588532 (+ 917us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:15.588612 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 02:50:15.588745 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28,"spinlock_wait_cycles":4352}
I20260806 02:50:15.589393 2574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.578490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:43615 (local address 127.0.0.1:46112)
0806 02:50:15.579038 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:15.579051 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:15.579157 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:15.579549 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:15.579551 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:15.579720 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:15.579815 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:15.579820 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.580658 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.580661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:15.581466 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:15.581474 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.582218 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.582221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:15.582318 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:15.582915 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:15.582926 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:15.585398 (+ 2472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:15.587379 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:15.587383 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:15.587384 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:15.587599 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:15.587922 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:15.587924 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:15.587926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:15.588022 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:15.588540 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:15.588542 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:15.588920 (+ 378us) client_negotiation.cc:769] Sending connection context
0806 02:50:15.589048 (+ 128us) client_negotiation.cc:240] Negotiation successful
0806 02:50:15.589206 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":264,"spinlock_wait_cycles":1792}
I20260806 02:50:15.589958 2365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.578936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42986 (local address 127.1.183.190:35499)
0806 02:50:15.579034 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:15.579038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:15.579220 (+ 182us) server_negotiation.cc:407] Connection header received
0806 02:50:15.579393 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:15.579398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:15.579467 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:15.579544 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:15.580342 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.580900 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.581683 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.581996 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.582104 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:15.585406 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:15.585409 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:15.585412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:15.585438 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:15.588014 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:15.588705 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:15.588711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:15.588713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:15.588793 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:15.589113 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:15.589116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:15.589118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:15.589297 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:15.589552 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:15.589668 (+ 116us) server_negotiation.cc:299] Negotiation successful
0806 02:50:15.589783 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":39,"spinlock_wait_cycles":36608}
I20260806 02:50:15.590608 2475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.578822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46112 (local address 127.1.183.189:43615)
0806 02:50:15.578955 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:15.578962 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:15.579096 (+ 134us) server_negotiation.cc:407] Connection header received
0806 02:50:15.579266 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:15.579269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:15.579323 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:15.579408 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:15.579937 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.580512 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.581626 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.581983 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.582091 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:15.585561 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:15.585566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:15.585570 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:15.585605 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:15.587248 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:15.587716 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:15.587720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:15.587725 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:15.587817 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:15.588120 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:15.588123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:15.588125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:15.588392 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:15.588495 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:15.590310 (+ 1815us) server_negotiation.cc:299] Negotiation successful
0806 02:50:15.590456 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 02:50:15.592270 2568 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35253 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:50:15.593196 2358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4ff79998ad0412bb554589b7b9d385e: Bootstrap starting.
Aug 06 02:50:15 dist-test-slave-gmjp krb5kdc[2237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984615, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.189@KRBTEST.COM for kudu/127.1.183.188@KRBTEST.COM
I20260806 02:50:15.596400 2358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4ff79998ad0412bb554589b7b9d385e: Neither blocks nor log segments found. Creating new log.
Aug 06 02:50:15 dist-test-slave-gmjp krb5kdc[2237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984615, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.188@KRBTEST.COM for kudu/127.1.183.188@KRBTEST.COM
I20260806 02:50:15.597410 2358 log.cc:824] T 00000000000000000000000000000000 P c4ff79998ad0412bb554589b7b9d385e: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:15.597625 2468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.587089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:35253 (local address 127.0.0.1:51068)
0806 02:50:15.587308 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:15.587326 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:15.587399 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:15.588109 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:15.588112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:15.588325 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:15.588415 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:15.588424 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.590517 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.590520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:15.591269 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:15.591278 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.591743 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.591747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:15.591832 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:15.592359 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:15.592373 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:15.593872 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:15.595909 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:15.595913 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:15.595915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:15.596198 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:15.596591 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:15.596598 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:15.596601 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:15.596701 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:15.597103 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:15.597106 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:15.597205 (+ 99us) client_negotiation.cc:769] Sending connection context
0806 02:50:15.597302 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 02:50:15.597450 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260806 02:50:15.598057 2576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.587179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51068 (local address 127.1.183.188:35253)
0806 02:50:15.587848 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:15.587859 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:15.587885 (+ 26us) server_negotiation.cc:407] Connection header received
0806 02:50:15.587920 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:15.587922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:15.587968 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:15.588528 (+ 560us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:15.588665 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.590389 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.591388 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.591626 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.591696 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:15.594026 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:15.594030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:15.594032 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:15.594056 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:15.595781 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:15.596335 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:15.596340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:15.596342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:15.596451 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:15.596812 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:15.596815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:15.596817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:15.597001 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:15.597110 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:15.597369 (+ 259us) server_negotiation.cc:299] Negotiation successful
0806 02:50:15.597578 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":596,"thread_start_us":135,"threads_started":1}
I20260806 02:50:15.600915 2358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4ff79998ad0412bb554589b7b9d385e: No bootstrap required, opened a new log
I20260806 02:50:15.600936 2574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.592779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:35253 (local address 127.0.0.1:51080)
0806 02:50:15.593103 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:15.593117 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:15.593182 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:15.593565 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:15.593568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:15.593736 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:15.593803 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:15.593808 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.594585 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.594588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:15.595228 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:15.595237 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.595615 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.595619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:15.595690 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:15.596119 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:15.596133 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:15.597550 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:15.599537 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:15.599540 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:15.599542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:15.599756 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:15.600008 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:15.600011 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:15.600013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:15.600091 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:15.600515 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:15.600517 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:15.600601 (+ 84us) client_negotiation.cc:769] Sending connection context
0806 02:50:15.600682 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 02:50:15.600799 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241}
I20260806 02:50:15.601409 2575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.592984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51080 (local address 127.1.183.188:35253)
0806 02:50:15.593384 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:15.593388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:15.593400 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:50:15.593436 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:15.593438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:15.593467 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:15.593555 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:15.593915 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.594473 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.595325 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.595511 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.595615 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:15.597679 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:15.597682 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:15.597685 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:15.597708 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:15.599436 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:15.599853 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:15.599856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:15.599858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:15.599924 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:15.600215 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:15.600217 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:15.600219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:15.600402 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:15.601091 (+ 689us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:15.601173 (+ 82us) server_negotiation.cc:299] Negotiation successful
0806 02:50:15.601283 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265}
I20260806 02:50:15.601737 2464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6d28389e22246dc8c4374504701afa8: Bootstrap starting.
I20260806 02:50:15.604801 2464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6d28389e22246dc8c4374504701afa8: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:15.604776 2358 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4ff79998ad0412bb554589b7b9d385e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ff79998ad0412bb554589b7b9d385e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35499 } } peers { permanent_uuid: "b6d28389e22246dc8c4374504701afa8" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 43615 } } peers { permanent_uuid: "f3fbe2801b9d4d219600e08936c8ad79" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35253 } }
I20260806 02:50:15.605067 2358 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4ff79998ad0412bb554589b7b9d385e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4ff79998ad0412bb554589b7b9d385e, State: Initialized, Role: FOLLOWER
I20260806 02:50:15.605377 2358 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4ff79998ad0412bb554589b7b9d385e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ff79998ad0412bb554589b7b9d385e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35499 } } peers { permanent_uuid: "b6d28389e22246dc8c4374504701afa8" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 43615 } } peers { permanent_uuid: "f3fbe2801b9d4d219600e08936c8ad79" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35253 } }
I20260806 02:50:15.605700 2464 log.cc:824] T 00000000000000000000000000000000 P b6d28389e22246dc8c4374504701afa8: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:15.605927 2582 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4ff79998ad0412bb554589b7b9d385e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ff79998ad0412bb554589b7b9d385e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35499 } } peers { permanent_uuid: "b6d28389e22246dc8c4374504701afa8" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 43615 } } peers { permanent_uuid: "f3fbe2801b9d4d219600e08936c8ad79" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35253 } } }
I20260806 02:50:15.606120 2582 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4ff79998ad0412bb554589b7b9d385e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:50:15.606727 2358 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4ff79998ad0412bb554589b7b9d385e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:15.607275 2568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3fbe2801b9d4d219600e08936c8ad79: Bootstrap starting.
I20260806 02:50:15.607872 2464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6d28389e22246dc8c4374504701afa8: No bootstrap required, opened a new log
I20260806 02:50:15.609948 2568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3fbe2801b9d4d219600e08936c8ad79: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:15.610855 2568 log.cc:824] T 00000000000000000000000000000000 P f3fbe2801b9d4d219600e08936c8ad79: Log is configured to *not* fsync() on all Append() calls
Aug 06 02:50:15 dist-test-slave-gmjp krb5kdc[2237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.189@KRBTEST.COM
I20260806 02:50:15.612097 2464 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6d28389e22246dc8c4374504701afa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ff79998ad0412bb554589b7b9d385e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35499 } } peers { permanent_uuid: "b6d28389e22246dc8c4374504701afa8" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 43615 } } peers { permanent_uuid: "f3fbe2801b9d4d219600e08936c8ad79" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35253 } }
I20260806 02:50:15.612365 2464 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6d28389e22246dc8c4374504701afa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6d28389e22246dc8c4374504701afa8, State: Initialized, Role: FOLLOWER
W20260806 02:50:15.612613 2601 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c4ff79998ad0412bb554589b7b9d385e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:50:15.612723 2601 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:50:15.612720 2464 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6d28389e22246dc8c4374504701afa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ff79998ad0412bb554589b7b9d385e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35499 } } peers { permanent_uuid: "b6d28389e22246dc8c4374504701afa8" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 43615 } } peers { permanent_uuid: "f3fbe2801b9d4d219600e08936c8ad79" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35253 } }
I20260806 02:50:15.614133 2606 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6d28389e22246dc8c4374504701afa8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ff79998ad0412bb554589b7b9d385e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35499 } } peers { permanent_uuid: "b6d28389e22246dc8c4374504701afa8" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 43615 } } peers { permanent_uuid: "f3fbe2801b9d4d219600e08936c8ad79" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35253 } } }
I20260806 02:50:15.614342 2606 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6d28389e22246dc8c4374504701afa8 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:50:15.614660 2568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3fbe2801b9d4d219600e08936c8ad79: No bootstrap required, opened a new log
I20260806 02:50:15.611971 2255 master_runner.cc:186] Error getting master registration for 127.1.183.189:43615: OK, instance_id { permanent_uuid: "b6d28389e22246dc8c4374504701afa8" instance_seqno: 1785984615328414 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 02:50:15.615824 2255 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:50:15.616468 2464 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6d28389e22246dc8c4374504701afa8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:15.619117 2568 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3fbe2801b9d4d219600e08936c8ad79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ff79998ad0412bb554589b7b9d385e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35499 } } peers { permanent_uuid: "b6d28389e22246dc8c4374504701afa8" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 43615 } } peers { permanent_uuid: "f3fbe2801b9d4d219600e08936c8ad79" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35253 } }
I20260806 02:50:15.619366 2568 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3fbe2801b9d4d219600e08936c8ad79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3fbe2801b9d4d219600e08936c8ad79, State: Initialized, Role: FOLLOWER
I20260806 02:50:15.619709 2568 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3fbe2801b9d4d219600e08936c8ad79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ff79998ad0412bb554589b7b9d385e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35499 } } peers { permanent_uuid: "b6d28389e22246dc8c4374504701afa8" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 43615 } } peers { permanent_uuid: "f3fbe2801b9d4d219600e08936c8ad79" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35253 } }
I20260806 02:50:15.620227 2475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.597916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46114 (local address 127.1.183.189:43615)
0806 02:50:15.598270 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:15.598274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:15.598288 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:50:15.598325 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:15.598328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:15.598379 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:15.598475 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:15.600261 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.600954 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.602430 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.602756 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.602870 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:15.612673 (+ 9803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:15.612678 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:15.612681 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:15.612707 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:15.614574 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:15.615147 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:15.615151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:15.615153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:15.615239 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:15.619043 (+ 3804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:15.619047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:15.619049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:15.619291 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:15.619514 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:15.619910 (+ 396us) server_negotiation.cc:299] Negotiation successful
0806 02:50:15.620055 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258}
I20260806 02:50:15.621150 2568 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3fbe2801b9d4d219600e08936c8ad79 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 02:50:15.622486 2624 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b6d28389e22246dc8c4374504701afa8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:50:15.622615 2624 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:50:15.625944 1758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260806 02:50:15.626704 2649 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f3fbe2801b9d4d219600e08936c8ad79: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:50:15.626785 2649 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:50:15.627068 2625 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3fbe2801b9d4d219600e08936c8ad79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ff79998ad0412bb554589b7b9d385e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35499 } } peers { permanent_uuid: "b6d28389e22246dc8c4374504701afa8" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 43615 } } peers { permanent_uuid: "f3fbe2801b9d4d219600e08936c8ad79" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35253 } } }
I20260806 02:50:15.627188 2625 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3fbe2801b9d4d219600e08936c8ad79 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:50:15.627655 2317 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c4ff79998ad0412bb554589b7b9d385e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42986
I20260806 02:50:15.627911 2317 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4ff79998ad0412bb554589b7b9d385e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 02:50:15.628050 2317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4ff79998ad0412bb554589b7b9d385e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:15.629511 2317 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4ff79998ad0412bb554589b7b9d385e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ff79998ad0412bb554589b7b9d385e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35499 } } peers { permanent_uuid: "b6d28389e22246dc8c4374504701afa8" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 43615 } } peers { permanent_uuid: "f3fbe2801b9d4d219600e08936c8ad79" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35253 } }
I20260806 02:50:15.631067 2428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c4ff79998ad0412bb554589b7b9d385e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b6d28389e22246dc8c4374504701afa8"
I20260806 02:50:15.631296 2428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6d28389e22246dc8c4374504701afa8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:15.631446 2533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c4ff79998ad0412bb554589b7b9d385e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f3fbe2801b9d4d219600e08936c8ad79"
I20260806 02:50:15.631662 2533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3fbe2801b9d4d219600e08936c8ad79 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:15.633023 2533 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f3fbe2801b9d4d219600e08936c8ad79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4ff79998ad0412bb554589b7b9d385e in term 1.
I20260806 02:50:15.633230 2428 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b6d28389e22246dc8c4374504701afa8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4ff79998ad0412bb554589b7b9d385e in term 1.
I20260806 02:50:15.633841 2284 leader_election.cc:302] T 00000000000000000000000000000000 P c4ff79998ad0412bb554589b7b9d385e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6d28389e22246dc8c4374504701afa8, c4ff79998ad0412bb554589b7b9d385e; no voters:
I20260806 02:50:15.634145 2582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4ff79998ad0412bb554589b7b9d385e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:15.634410 2582 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4ff79998ad0412bb554589b7b9d385e [term 1 LEADER]: Becoming Leader. State: Replica: c4ff79998ad0412bb554589b7b9d385e, State: Running, Role: LEADER
I20260806 02:50:15.634775 2582 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4ff79998ad0412bb554589b7b9d385e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ff79998ad0412bb554589b7b9d385e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35499 } } peers { permanent_uuid: "b6d28389e22246dc8c4374504701afa8" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 43615 } } peers { permanent_uuid: "f3fbe2801b9d4d219600e08936c8ad79" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35253 } }
I20260806 02:50:15.635604 2370 master_runner.cc:232] Existing masters have differing terms: 0 vs 1
I20260806 02:50:15.635721 2370 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:50:15.635807 2476 master_runner.cc:232] Existing masters have differing terms: 0 vs 1
I20260806 02:50:15.635910 2476 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:50:15.635983 2317 leader_election.cc:288] T 00000000000000000000000000000000 P c4ff79998ad0412bb554589b7b9d385e [CANDIDATE]: Term 1 election: Requested vote from peers b6d28389e22246dc8c4374504701afa8 (127.1.183.189:43615), f3fbe2801b9d4d219600e08936c8ad79 (127.1.183.188:35253)
I20260806 02:50:15.636704 2582 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4ff79998ad0412bb554589b7b9d385e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4ff79998ad0412bb554589b7b9d385e. Latest consensus state: current_term: 1 leader_uuid: "c4ff79998ad0412bb554589b7b9d385e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ff79998ad0412bb554589b7b9d385e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35499 } } peers { permanent_uuid: "b6d28389e22246dc8c4374504701afa8" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 43615 } } peers { permanent_uuid: "f3fbe2801b9d4d219600e08936c8ad79" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35253 } } }
I20260806 02:50:15.636823 2582 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4ff79998ad0412bb554589b7b9d385e [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:15.648919 2575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.638222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51092 (local address 127.1.183.188:35253)
0806 02:50:15.638339 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:15.638344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:15.638955 (+ 611us) server_negotiation.cc:407] Connection header received
0806 02:50:15.639151 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:15.639154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:15.639203 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:15.639254 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:15.640379 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.641187 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.642408 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.642759 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.642838 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:15.644915 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:15.644921 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:15.644923 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:15.644952 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:15.646779 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:15.647306 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:15.647310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:15.647312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:15.647402 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:15.647758 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:15.647762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:15.647763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:15.648039 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:15.648199 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:15.648608 (+ 409us) server_negotiation.cc:299] Negotiation successful
0806 02:50:15.648741 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 02:50:15.658823 2654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:50:15.660035 2475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.645263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46120 (local address 127.1.183.189:43615)
0806 02:50:15.645402 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:15.645406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:15.650743 (+ 5337us) server_negotiation.cc:407] Connection header received
0806 02:50:15.650932 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:15.650936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:15.650992 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:15.651047 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:15.651990 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.652752 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.653983 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.654307 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.654380 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:15.656242 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:15.656246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:15.656250 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:15.656283 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:15.658022 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:15.658566 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:15.658569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:15.658571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:15.658665 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:15.658999 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:15.659003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:15.659004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:15.659210 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:15.659315 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:15.659700 (+ 385us) server_negotiation.cc:299] Negotiation successful
0806 02:50:15.659821 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 02:50:15.663792 2365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:15.644553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42998 (local address 127.1.183.190:35499)
0806 02:50:15.644679 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:15.644683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:15.645855 (+ 1172us) server_negotiation.cc:407] Connection header received
0806 02:50:15.652361 (+ 6506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:15.652365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:15.652416 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:15.652467 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:15.653392 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.654108 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.655202 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:15.655562 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:15.655663 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:15.659700 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:15.659705 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:15.659709 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:15.659735 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:15.661499 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:15.662178 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:15.662183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:15.662185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:15.662306 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:15.662642 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:15.662645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:15.662647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:15.662851 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:15.662947 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:15.663483 (+ 536us) server_negotiation.cc:299] Negotiation successful
0806 02:50:15.663604 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:50:15.663635 2654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:15.669486 2428 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b6d28389e22246dc8c4374504701afa8 [term 1 FOLLOWER]: Refusing update from remote peer c4ff79998ad0412bb554589b7b9d385e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:15.670287 2533 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f3fbe2801b9d4d219600e08936c8ad79 [term 1 FOLLOWER]: Refusing update from remote peer c4ff79998ad0412bb554589b7b9d385e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:15.671371 2582 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c4ff79998ad0412bb554589b7b9d385e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6d28389e22246dc8c4374504701afa8" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 43615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:15.672094 2582 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c4ff79998ad0412bb554589b7b9d385e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3fbe2801b9d4d219600e08936c8ad79" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:15.675845 2625 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3fbe2801b9d4d219600e08936c8ad79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4ff79998ad0412bb554589b7b9d385e. Latest consensus state: current_term: 1 leader_uuid: "c4ff79998ad0412bb554589b7b9d385e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ff79998ad0412bb554589b7b9d385e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35499 } } peers { permanent_uuid: "b6d28389e22246dc8c4374504701afa8" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 43615 } } peers { permanent_uuid: "f3fbe2801b9d4d219600e08936c8ad79" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35253 } } }
I20260806 02:50:15.675989 2625 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3fbe2801b9d4d219600e08936c8ad79 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:50:15.681198 2606 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6d28389e22246dc8c4374504701afa8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4ff79998ad0412bb554589b7b9d385e. Latest consensus state: current_term: 1 leader_uuid: "c4ff79998ad0412bb554589b7b9d385e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ff79998ad0412bb554589b7b9d385e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35499 } } peers { permanent_uuid: "b6d28389e22246dc8c4374504701afa8" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 43615 } } peers { permanent_uuid: "f3fbe2801b9d4d219600e08936c8ad79" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35253 } } }
I20260806 02:50:15.681347 2606 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6d28389e22246dc8c4374504701afa8 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:50:15.688728 2582 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4ff79998ad0412bb554589b7b9d385e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c4ff79998ad0412bb554589b7b9d385e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ff79998ad0412bb554589b7b9d385e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35499 } } peers { permanent_uuid: "b6d28389e22246dc8c4374504701afa8" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 43615 } } peers { permanent_uuid: "f3fbe2801b9d4d219600e08936c8ad79" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35253 } } }
I20260806 02:50:15.688879 2582 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4ff79998ad0412bb554589b7b9d385e [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:15.689556 2668 mvcc.cc:205] Tried to move back new op lower bound from 7315392985774952448 to 7315392985643745280. Current Snapshot: MvccSnapshot[applied={T|T < 7315392985774952448}]
I20260806 02:50:15.691619 2625 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3fbe2801b9d4d219600e08936c8ad79 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c4ff79998ad0412bb554589b7b9d385e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ff79998ad0412bb554589b7b9d385e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35499 } } peers { permanent_uuid: "b6d28389e22246dc8c4374504701afa8" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 43615 } } peers { permanent_uuid: "f3fbe2801b9d4d219600e08936c8ad79" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35253 } } }
I20260806 02:50:15.691751 2625 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3fbe2801b9d4d219600e08936c8ad79 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:50:15.696573 2652 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4ff79998ad0412bb554589b7b9d385e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c4ff79998ad0412bb554589b7b9d385e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ff79998ad0412bb554589b7b9d385e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35499 } } peers { permanent_uuid: "b6d28389e22246dc8c4374504701afa8" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 43615 } } peers { permanent_uuid: "f3fbe2801b9d4d219600e08936c8ad79" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35253 } } }
I20260806 02:50:15.696743 2652 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4ff79998ad0412bb554589b7b9d385e [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:15.699891 2606 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6d28389e22246dc8c4374504701afa8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c4ff79998ad0412bb554589b7b9d385e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ff79998ad0412bb554589b7b9d385e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 35499 } } peers { permanent_uuid: "b6d28389e22246dc8c4374504701afa8" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 43615 } } peers { permanent_uuid: "f3fbe2801b9d4d219600e08936c8ad79" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 35253 } } }
I20260806 02:50:15.700033 2606 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6d28389e22246dc8c4374504701afa8 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:50:15.696679 2654 catalog_manager.cc:1347] Generated new cluster ID: 999cd8f19b9747a39be6c520bbbcbd37
I20260806 02:50:15.701001 2654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:15.716503 2671 mvcc.cc:205] Tried to move back new op lower bound from 7315392985774952448 to 7315392985643745280. Current Snapshot: MvccSnapshot[applied={T|T < 7315392985774952448}]
I20260806 02:50:15.730511 2654 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:15.733889 2654 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:15.754648 2654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4ff79998ad0412bb554589b7b9d385e: Generated new TSK 0
I20260806 02:50:15.755612 2654 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:50:15.794796 2365 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:15.791218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43002 (local address 127.1.183.190:35499)
0806 02:50:15.791362 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:15.791366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:15.793581 (+ 2215us) server_negotiation.cc:407] Connection header received
0806 02:50:15.793819 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:15.793823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:15.793881 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:15.793940 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:15.794642 (+ 702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43002: BlockingRecv error: recv got EOF from 127.0.0.1:43002 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260806 02:50:15.796283 2475 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:15.792038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46124 (local address 127.1.183.189:43615)
0806 02:50:15.792182 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:15.792188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:15.795326 (+ 3138us) server_negotiation.cc:407] Connection header received
0806 02:50:15.795468 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:15.795471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:15.795523 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:15.795576 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:15.796114 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46124: BlockingRecv error: recv got EOF from 127.0.0.1:46124 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260806 02:50:15.797847 2575 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:15.792890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51108 (local address 127.1.183.188:35253)
0806 02:50:15.793025 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:15.793030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:15.796787 (+ 3757us) server_negotiation.cc:407] Connection header received
0806 02:50:15.796971 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:15.796974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:15.797027 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:15.797083 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:15.797700 (+ 617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51108: BlockingRecv error: recv got EOF from 127.0.0.1:51108 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260806 02:50:15.802783 2475 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:15.801118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46132 (local address 127.1.183.189:43615)
0806 02:50:15.801232 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:15.801236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:15.801753 (+ 517us) server_negotiation.cc:407] Connection header received
0806 02:50:15.801995 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:15.802001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:15.802080 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:15.802134 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:15.802658 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46132: BlockingRecv error: recv got EOF from 127.0.0.1:46132 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260806 02:50:15.808528 2475 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:15.806681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46144 (local address 127.1.183.189:43615)
0806 02:50:15.806804 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:15.806808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:15.807330 (+ 522us) server_negotiation.cc:407] Connection header received
0806 02:50:15.807531 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:15.807534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:15.807582 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:15.807630 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:15.808420 (+ 790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46144: BlockingRecv error: recv got EOF from 127.0.0.1:46144 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260806 02:50:15.814735 2475 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:15.813110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46148 (local address 127.1.183.189:43615)
0806 02:50:15.813217 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:15.813221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:15.813742 (+ 521us) server_negotiation.cc:407] Connection header received
0806 02:50:15.813960 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:15.813964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:15.814015 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:15.814069 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:15.814621 (+ 552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46148: BlockingRecv error: recv got EOF from 127.0.0.1:46148 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260806 02:50:15.825901 2475 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:15.823950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46160 (local address 127.1.183.189:43615)
0806 02:50:15.824111 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:15.824115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:15.824815 (+ 700us) server_negotiation.cc:407] Connection header received
0806 02:50:15.825047 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:15.825051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:15.825112 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:15.825172 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:15.825784 (+ 612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46160: BlockingRecv error: recv got EOF from 127.0.0.1:46160 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260806 02:50:15.838418 2475 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:15.836679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46166 (local address 127.1.183.189:43615)
0806 02:50:15.836815 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:15.836820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:15.837426 (+ 606us) server_negotiation.cc:407] Connection header received
0806 02:50:15.837629 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:15.837633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:15.837686 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:15.837737 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:15.838298 (+ 561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46166: BlockingRecv error: recv got EOF from 127.0.0.1:46166 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260806 02:50:15.858703 2475 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:15.857515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46172 (local address 127.1.183.189:43615)
0806 02:50:15.857688 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:15.857693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:15.857839 (+ 146us) server_negotiation.cc:407] Connection header received
0806 02:50:15.857955 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:15.857959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:15.858023 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:15.858138 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:15.858555 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46172: BlockingRecv error: recv got EOF from 127.0.0.1:46172 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260806 02:50:15.894474 2475 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:15.893185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46176 (local address 127.1.183.189:43615)
0806 02:50:15.893364 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:15.893369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:15.893503 (+ 134us) server_negotiation.cc:407] Connection header received
0806 02:50:15.893695 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:15.893699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:15.893748 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:15.893845 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:15.894360 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46176: BlockingRecv error: recv got EOF from 127.0.0.1:46176 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260806 02:50:15.962082 2475 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:15.960808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46192 (local address 127.1.183.189:43615)
0806 02:50:15.960980 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:15.960986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:15.961124 (+ 138us) server_negotiation.cc:407] Connection header received
0806 02:50:15.961321 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:15.961324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:15.961372 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:15.961482 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:15.961970 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46192: BlockingRecv error: recv got EOF from 127.0.0.1:46192 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260806 02:50:16.093833 2475 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:16.092539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46196 (local address 127.1.183.189:43615)
0806 02:50:16.092714 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:16.092720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:16.092864 (+ 144us) server_negotiation.cc:407] Connection header received
0806 02:50:16.093059 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:16.093062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:16.093121 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:16.093221 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:16.093714 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46196: BlockingRecv error: recv got EOF from 127.0.0.1:46196 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260806 02:50:16.353665 2475 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:16.352395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46208 (local address 127.1.183.189:43615)
0806 02:50:16.352574 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:16.352579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:16.352707 (+ 128us) server_negotiation.cc:407] Connection header received
0806 02:50:16.352890 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:16.352893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:16.352950 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:16.353046 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:16.353561 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46208: BlockingRecv error: recv got EOF from 127.0.0.1:46208 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 02:50:16.624086 2624 catalog_manager.cc:1259] Loaded cluster ID: 999cd8f19b9747a39be6c520bbbcbd37
I20260806 02:50:16.624275 2624 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b6d28389e22246dc8c4374504701afa8: loading cluster ID for follower catalog manager: success
I20260806 02:50:16.625968 2624 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b6d28389e22246dc8c4374504701afa8: acquiring CA information for follower catalog manager: success
I20260806 02:50:16.627244 2624 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b6d28389e22246dc8c4374504701afa8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:50:16.628070 2649 catalog_manager.cc:1259] Loaded cluster ID: 999cd8f19b9747a39be6c520bbbcbd37
I20260806 02:50:16.628158 2649 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f3fbe2801b9d4d219600e08936c8ad79: loading cluster ID for follower catalog manager: success
I20260806 02:50:16.629745 2649 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f3fbe2801b9d4d219600e08936c8ad79: acquiring CA information for follower catalog manager: success
I20260806 02:50:16.630970 2649 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f3fbe2801b9d4d219600e08936c8ad79: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:50:16.872362 2790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:16.868113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46212 (local address 127.1.183.189:43615)
0806 02:50:16.868438 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:16.868442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:16.868598 (+ 156us) server_negotiation.cc:407] Connection header received
0806 02:50:16.868773 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:16.868779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:16.868980 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:16.869108 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:50:16.869360 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:16.869896 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:16.870768 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:16.871006 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:16.871099 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:16.871405 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:50:16.871863 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:50:16.871963 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:16.872109 (+ 146us) server_negotiation.cc:299] Negotiation successful
0806 02:50:16.872221 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":85,"threads_started":1}
I20260806 02:50:16.879846 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:16.875906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51112 (local address 127.1.183.188:35253)
0806 02:50:16.876142 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:16.876146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:16.876207 (+ 61us) server_negotiation.cc:407] Connection header received
0806 02:50:16.876370 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:16.876373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:16.876496 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:16.876605 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:50:16.876904 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:16.877418 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:16.878262 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:16.878490 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:16.878575 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:16.878968 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:50:16.879475 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:50:16.879580 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:16.879663 (+ 83us) server_negotiation.cc:299] Negotiation successful
0806 02:50:16.879714 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":84,"threads_started":1}
I20260806 02:50:16.887434 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:16.883062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51124 (local address 127.1.183.188:35253)
0806 02:50:16.883183 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:16.883188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:16.883325 (+ 137us) server_negotiation.cc:407] Connection header received
0806 02:50:16.883496 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:16.883500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:16.883551 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:16.883621 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:50:16.884020 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:16.884995 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:16.886188 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:16.886521 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:16.886621 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:16.886876 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:50:16.886976 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:50:16.887041 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:16.887201 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 02:50:16.887262 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 02:50:16.887812 2790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:16.883551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46214 (local address 127.1.183.189:43615)
0806 02:50:16.883706 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:16.883710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:16.883844 (+ 134us) server_negotiation.cc:407] Connection header received
0806 02:50:16.883934 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:16.883936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:16.883996 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:16.884075 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:50:16.884365 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:16.884921 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:16.886567 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:16.886914 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:16.887034 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:16.887300 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:50:16.887399 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:50:16.887444 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:16.887617 (+ 173us) server_negotiation.cc:299] Negotiation successful
0806 02:50:16.887675 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 02:50:16.888047 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:16.883220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43010 (local address 127.1.183.190:35499)
0806 02:50:16.883494 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:16.883499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:16.883512 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:50:16.883624 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:16.883628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:16.883784 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:16.883945 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:50:16.884221 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:16.885109 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:16.886017 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:16.886364 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:16.886468 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:16.886805 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:50:16.887194 (+ 389us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:50:16.887282 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:16.887822 (+ 540us) server_negotiation.cc:299] Negotiation successful
0806 02:50:16.887903 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":69,"threads_started":1}
I20260806 02:50:16.894909 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 2255
I20260806 02:50:16.907612 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 2370
I20260806 02:50:16.924266 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 2476
2026-08-06T02:50:16Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3389 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:50:16 dist-test-slave-gmjp krb5kdc[2814](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:50:16 dist-test-slave-gmjp krb5kdc[2814](info): set up 2 sockets
Aug 06 02:50:16 dist-test-slave-gmjp krb5kdc[2814](info): commencing operation
krb5kdc: starting...
W20260806 02:50:18.165267 1758 mini_kdc.cc:121] Time spent starting KDC: real 1.216s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/krb5kdc/test-user.keytab.
Aug 06 02:50:18 dist-test-slave-gmjp krb5kdc[2814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:50:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:50:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.183.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
Entry for principal kudu/127.1.183.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab.
I20260806 02:50:18.299571 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:34181
--webserver_interface=127.1.183.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:45647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:34181,127.1.183.189:33983,127.1.183.188:44889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:50:18.414362 2828 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:50:18.414760 2828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:18.414834 2828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:18.414915 2828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:18.419045 2828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:18.419155 2828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:18.419191 2828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:18.419234 2828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:18.419265 2828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:18.424789 2828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:45647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:34181,127.1.183.189:33983,127.1.183.188:44889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.190.keytab
--principal=kudu/127.1.183.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.190:34181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.183.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:18.426031 2828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:18.427134 2828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:18.436067 2836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:18.436286 2828 server_base.cc:1034] running on GCE node
W20260806 02:50:18.436071 2838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:18.436067 2835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:18.437021 2828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:18.438155 2828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:18.439328 2828 hybrid_clock.cc:648] HybridClock initialized: now 1785984618439312 us; error 32 us; skew 500 ppm
Aug 06 02:50:18 dist-test-slave-gmjp krb5kdc[2814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984618, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:18.442193 2828 init.cc:377] Logged in from keytab as kudu/127.1.183.190@KRBTEST.COM (short username kudu)
I20260806 02:50:18.443167 2828 webserver.cc:460] Webserver started at http://127.1.183.190:35069/ using document root <none> and password file <none>
I20260806 02:50:18.443711 2828 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:18.443771 2828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:18.443945 2828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:18.446094 2828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-0/data/instance:
uuid: "bdd7b9822ef2416ea8061a77c178ee2e"
format_stamp: "Formatted at 2026-08-06 02:50:18 on dist-test-slave-gmjp"
server_key: "9ac8a28403879f8920b4d426c7cbbeb9"
server_key_iv: "2241f8a0f5fae4da30ad4f691d39a649"
server_key_version: "encryptionkey@0"
I20260806 02:50:18.446666 2828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-0/wal/instance:
uuid: "bdd7b9822ef2416ea8061a77c178ee2e"
format_stamp: "Formatted at 2026-08-06 02:50:18 on dist-test-slave-gmjp"
server_key: "9ac8a28403879f8920b4d426c7cbbeb9"
server_key_iv: "2241f8a0f5fae4da30ad4f691d39a649"
server_key_version: "encryptionkey@0"
I20260806 02:50:18.450668 2828 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:50:18.453112 2849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:18.454363 2828 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:50:18.454533 2828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-0/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-0/wal
uuid: "bdd7b9822ef2416ea8061a77c178ee2e"
format_stamp: "Formatted at 2026-08-06 02:50:18 on dist-test-slave-gmjp"
server_key: "9ac8a28403879f8920b4d426c7cbbeb9"
server_key_iv: "2241f8a0f5fae4da30ad4f691d39a649"
server_key_version: "encryptionkey@0"
I20260806 02:50:18.454634 2828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:18.470530 2828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:18.473776 2828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:18.474011 2828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:18.481604 2828 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.190:34181
I20260806 02:50:18.481931 2921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.190:34181 every 8 connection(s)
I20260806 02:50:18.482667 2828 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-0/data/info.pb
I20260806 02:50:18.485507 2922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:18.486987 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 2828
I20260806 02:50:18.487143 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-0/wal/instance
I20260806 02:50:18.487445 1758 external_mini_cluster.cc:1428] Setting key b0e288ae29adb5a30a9efe0cede19493
I20260806 02:50:18.489540 2922 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34181 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.183.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.189@KRBTEST.COM" created.
Aug 06 02:50:18 dist-test-slave-gmjp krb5kdc[2814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984618, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:50:18.505733 2930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.491922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34181 (local address 127.0.0.1:48478)
0806 02:50:18.492565 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:18.492624 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:18.493466 (+ 842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:18.494159 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:18.494164 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:18.494626 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:18.494862 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:18.494876 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.496022 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.496025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:18.496862 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:18.496873 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.497538 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.497541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:18.497695 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:18.498211 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:18.498227 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:18.500317 (+ 2090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:18.502773 (+ 2456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:18.502782 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:18.502790 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:18.503013 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:18.503409 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:18.503411 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:18.503412 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:18.503517 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:18.504387 (+ 870us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:18.504392 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:18.504519 (+ 127us) client_negotiation.cc:769] Sending connection context
0806 02:50:18.504655 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 02:50:18.504915 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":41,"spinlock_wait_cycles":22784,"thread_start_us":209,"threads_started":1}
I20260806 02:50:18.505738 2927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.491922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48478 (local address 127.1.183.190:34181)
0806 02:50:18.492356 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:18.492376 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:18.492746 (+ 370us) server_negotiation.cc:407] Connection header received
0806 02:50:18.493772 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:18.493776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:18.493971 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:18.494161 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:18.495012 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.495891 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.496984 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.497376 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.497540 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:18.500492 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:18.500498 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:18.500508 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:18.500531 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:18.502651 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:18.503149 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:18.503158 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:18.503163 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:18.503311 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:18.503622 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:18.503629 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:18.503633 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:18.504248 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:18.504407 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:18.504806 (+ 399us) server_negotiation.cc:299] Negotiation successful
0806 02:50:18.504988 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":292,"thread_start_us":168,"threads_started":1}
I20260806 02:50:18.508375 2922 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.183.189" port: 33983 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:50:18.509107 2930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.508843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:33983 (local address 127.0.0.1:48502)
0806 02:50:18.508970 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:18.509048 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.189:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 02:50:18.509351 2859 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.183.189:33983: connect: Connection refused (error 111)
I20260806 02:50:18.510241 2930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.509991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:33983 (local address 127.0.0.1:48508)
0806 02:50:18.510119 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:18.510175 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.189:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 02:50:18.510409 2922 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.183.189:33983: Network error: Client connection negotiation failed: client connection to 127.1.183.189:33983: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.189.keytab.
Entry for principal kudu/127.1.183.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.189.keytab.
I20260806 02:50:18.517755 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.189:33983
--webserver_interface=127.1.183.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.189.keytab
--principal=kudu/127.1.183.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:45647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:34181,127.1.183.189:33983,127.1.183.188:44889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:50:18.559665 2922 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.183.189" port: 33983 } attempt: 1
I20260806 02:50:18.560616 2930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.560313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:33983 (local address 127.0.0.1:48524)
0806 02:50:18.560468 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:18.560534 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.189:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 02:50:18.561358 2930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.561126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:33983 (local address 127.0.0.1:48532)
0806 02:50:18.561240 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:18.561300 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.189:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 02:50:18.561493 2922 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.183.189:33983: Network error: Client connection negotiation failed: client connection to 127.1.183.189:33983: connect: Connection refused (error 111)
I20260806 02:50:18.629679 2922 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.183.189" port: 33983 } attempt: 2
I20260806 02:50:18.630623 2930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.630307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:33983 (local address 127.0.0.1:48548)
0806 02:50:18.630473 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:18.630544 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.189:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 02:50:18.631445 2930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.631243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:33983 (local address 127.0.0.1:48562)
0806 02:50:18.631347 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:18.631392 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.189:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 02:50:18.631621 2922 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.183.189:33983: Network error: Client connection negotiation failed: client connection to 127.1.183.189:33983: connect: Connection refused (error 111)
W20260806 02:50:18.632776 2938 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:50:18.633049 2938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:18.633102 2938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:18.633195 2938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:50:18.637255 2938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:18.637315 2938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:18.637334 2938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:18.637351 2938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:18.637404 2938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:18.642956 2938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:45647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:34181,127.1.183.189:33983,127.1.183.188:44889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.189.keytab
--principal=kudu/127.1.183.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.189:33983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.183.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:18.644235 2938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:18.645345 2938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:18.654158 2945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:18.654174 2950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:18.654181 2946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:18.654230 2938 server_base.cc:1034] running on GCE node
I20260806 02:50:18.655012 2938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:18.656124 2938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:18.657351 2938 hybrid_clock.cc:648] HybridClock initialized: now 1785984618657333 us; error 32 us; skew 500 ppm
Aug 06 02:50:18 dist-test-slave-gmjp krb5kdc[2814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984618, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:18.660367 2938 init.cc:377] Logged in from keytab as kudu/127.1.183.189@KRBTEST.COM (short username kudu)
I20260806 02:50:18.661267 2938 webserver.cc:460] Webserver started at http://127.1.183.189:45681/ using document root <none> and password file <none>
I20260806 02:50:18.661846 2938 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:18.661960 2938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:18.662192 2938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:18.664321 2938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-1/data/instance:
uuid: "a6eb502e44df419fb3d82290a5dc912b"
format_stamp: "Formatted at 2026-08-06 02:50:18 on dist-test-slave-gmjp"
server_key: "f7b2367ce07f8cab185e7e4ea4e11324"
server_key_iv: "c0aaf5e6dedd694ed1e6a2647ad00d8f"
server_key_version: "encryptionkey@0"
I20260806 02:50:18.664911 2938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-1/wal/instance:
uuid: "a6eb502e44df419fb3d82290a5dc912b"
format_stamp: "Formatted at 2026-08-06 02:50:18 on dist-test-slave-gmjp"
server_key: "f7b2367ce07f8cab185e7e4ea4e11324"
server_key_iv: "c0aaf5e6dedd694ed1e6a2647ad00d8f"
server_key_version: "encryptionkey@0"
I20260806 02:50:18.668758 2938 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:50:18.671415 2966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:18.672777 2938 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:50:18.672926 2938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-1/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-1/wal
uuid: "a6eb502e44df419fb3d82290a5dc912b"
format_stamp: "Formatted at 2026-08-06 02:50:18 on dist-test-slave-gmjp"
server_key: "f7b2367ce07f8cab185e7e4ea4e11324"
server_key_iv: "c0aaf5e6dedd694ed1e6a2647ad00d8f"
server_key_version: "encryptionkey@0"
I20260806 02:50:18.673041 2938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:18.689551 2938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:18.693169 2938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:18.693414 2938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:18.701169 2938 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.189:33983
I20260806 02:50:18.701561 3047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.189:33983 every 8 connection(s)
I20260806 02:50:18.702245 2938 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-1/data/info.pb
I20260806 02:50:18.705076 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 2938
I20260806 02:50:18.705231 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-1/wal/instance
I20260806 02:50:18.705449 3048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:18.705587 1758 external_mini_cluster.cc:1428] Setting key dd981c56ca55a681327454648ecb390e
I20260806 02:50:18.709942 3048 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34181 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.183.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.183.188@KRBTEST.COM" created.
Aug 06 02:50:18 dist-test-slave-gmjp krb5kdc[2814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984618, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.189@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:50:18.722783 2922 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.183.189" port: 33983 } attempt: 3
I20260806 02:50:18.724622 2927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.711944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48488 (local address 127.1.183.190:34181)
0806 02:50:18.712124 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:18.712130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:18.713060 (+ 930us) server_negotiation.cc:407] Connection header received
0806 02:50:18.714024 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:18.714028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:18.714084 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:18.714190 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:18.715539 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.716073 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.717395 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.717651 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.717781 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:18.720518 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:18.720521 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:18.720523 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:18.720550 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:18.722018 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:18.722599 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:18.722603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:18.722604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:18.722685 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:18.723201 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:18.723208 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:18.723212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:18.723468 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:18.723572 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:18.724302 (+ 730us) server_negotiation.cc:299] Negotiation successful
0806 02:50:18.724449 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 02:50:18.725586 3051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.712372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34181 (local address 127.0.0.1:48488)
0806 02:50:18.712857 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:18.712920 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:18.713784 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:18.714441 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:18.714448 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:18.714889 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:18.715380 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:18.715394 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.716270 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.716274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:18.717213 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:18.717228 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.717827 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.717832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:18.718021 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:18.718649 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:18.718666 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:18.720321 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:18.722155 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:18.722162 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:18.722180 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:18.722471 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:18.722788 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:18.722794 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:18.722798 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:18.723026 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:18.723628 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:18.723635 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:18.723989 (+ 354us) client_negotiation.cc:769] Sending connection context
0806 02:50:18.724212 (+ 223us) client_negotiation.cc:240] Negotiation successful
0806 02:50:18.724569 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":139,"threads_started":1}
I20260806 02:50:18.727219 3048 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.183.189" port: 33983 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:50:18 dist-test-slave-gmjp krb5kdc[2814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984618, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for kudu/127.1.183.189@KRBTEST.COM
Aug 06 02:50:18 dist-test-slave-gmjp krb5kdc[2814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984618, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.189@KRBTEST.COM for kudu/127.1.183.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.183.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.188.keytab.
Entry for principal kudu/127.1.183.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.188.keytab.
I20260806 02:50:18.735502 2930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.723314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:33983 (local address 127.0.0.1:48564)
0806 02:50:18.723533 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:18.723551 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:18.723645 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:18.724591 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:18.724595 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:18.724787 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:18.724939 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:18.724946 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.726567 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.726570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:18.727827 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:18.727840 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.728747 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.728749 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:18.728919 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:18.729423 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:18.729438 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:18.731081 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:18.733447 (+ 2366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:18.733451 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:18.733453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:18.733720 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:18.734097 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:18.734100 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:18.734102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:18.734197 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:18.734807 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:18.734813 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:18.734939 (+ 126us) client_negotiation.cc:769] Sending connection context
0806 02:50:18.735059 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 02:50:18.735234 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260806 02:50:18.736272 1758 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hWVsi/build/debug/bin/kudu
/tmp/dist-test-task0hWVsi/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.188:44889
--webserver_interface=127.1.183.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.188.keytab
--principal=kudu/127.1.183.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.183.148:45647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.183.190:34181,127.1.183.189:33983,127.1.183.188:44889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:50:18.736778 3055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.723566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48564 (local address 127.1.183.189:33983)
0806 02:50:18.724031 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:18.724046 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:18.724075 (+ 29us) server_negotiation.cc:407] Connection header received
0806 02:50:18.724129 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:18.724138 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:18.724421 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:18.725110 (+ 689us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:18.725239 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.726451 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.727971 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.728619 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.728733 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:18.731217 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:18.731227 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:18.731240 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:18.731263 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:18.733333 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:18.733837 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:18.733843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:18.733844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:18.733941 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:18.734312 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:18.734320 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:18.734321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:18.734683 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:18.734857 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:18.735986 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0806 02:50:18.736482 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":317,"spinlock_wait_cycles":29568,"thread_start_us":178,"threads_started":1}
I20260806 02:50:18.737843 3051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.727810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:33983 (local address 127.0.0.1:48572)
0806 02:50:18.727940 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:18.727955 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:18.728032 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:18.728542 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:18.728545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:18.728743 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:18.728933 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:18.728942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.730024 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.730027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:18.730879 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:18.730892 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.731458 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.731462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:18.731579 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:18.732025 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:18.732042 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:18.733453 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:18.735893 (+ 2440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:18.735896 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:18.735898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:18.736591 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:18.736909 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:18.736913 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:18.736916 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:18.737004 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:18.737342 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:18.737345 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:18.737439 (+ 94us) client_negotiation.cc:769] Sending connection context
0806 02:50:18.737543 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 02:50:18.737694 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":37}
I20260806 02:50:18.738867 3057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.727961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48572 (local address 127.1.183.189:33983)
0806 02:50:18.728335 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:18.728342 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:18.728353 (+ 11us) server_negotiation.cc:407] Connection header received
0806 02:50:18.728384 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:18.728386 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:18.728419 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:18.728488 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:18.729071 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.729869 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.731021 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.731339 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.731450 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:18.733613 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:18.733618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:18.733621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:18.733647 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:18.735788 (+ 2141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:18.736699 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:18.736705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:18.736707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:18.736792 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:18.737087 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:18.737089 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:18.737091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:18.737245 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:18.737351 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:18.737599 (+ 248us) server_negotiation.cc:299] Negotiation successful
0806 02:50:18.738682 (+ 1083us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":973,"server-negotiator.queue_time_us":319,"thread_start_us":201,"threads_started":1}
I20260806 02:50:18.738893 2922 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.183.188" port: 44889 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:50:18.739444 3048 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.183.188" port: 44889 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:50:18.739653 2930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.739369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:44889 (local address 127.0.0.1:36644)
0806 02:50:18.739553 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:18.739597 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:44889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260806 02:50:18.740130 3051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.739875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:44889 (local address 127.0.0.1:36656)
0806 02:50:18.739986 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:18.740063 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:44889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 02:50:18.740335 2973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.183.188:44889: connect: Connection refused (error 111)
I20260806 02:50:18.740783 2930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.740451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:44889 (local address 127.0.0.1:36662)
0806 02:50:18.740641 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:18.740704 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:44889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260806 02:50:18.741101 2922 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.183.188:44889: Network error: Client connection negotiation failed: client connection to 127.1.183.188:44889: connect: Connection refused (error 111)
I20260806 02:50:18.741261 3051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.740971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:44889 (local address 127.0.0.1:36678)
0806 02:50:18.741052 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:18.741199 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:44889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260806 02:50:18.741421 3048 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.183.188:44889: Network error: Client connection negotiation failed: client connection to 127.1.183.188:44889: connect: Connection refused (error 111)
I20260806 02:50:18.772265 2922 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.183.188" port: 44889 } attempt: 1
I20260806 02:50:18.773201 2930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.772810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:44889 (local address 127.0.0.1:36690)
0806 02:50:18.773003 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:18.773088 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:44889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 02:50:18.773969 2930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.773738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:44889 (local address 127.0.0.1:36694)
0806 02:50:18.773849 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:18.773905 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:44889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 02:50:18.774119 2922 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.183.188:44889: Network error: Client connection negotiation failed: client connection to 127.1.183.188:44889: connect: Connection refused (error 111)
I20260806 02:50:18.790678 3048 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.183.188" port: 44889 } attempt: 1
I20260806 02:50:18.791635 3051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.791324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:44889 (local address 127.0.0.1:36710)
0806 02:50:18.791490 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:18.791553 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:44889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 02:50:18.792469 3051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.792219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:44889 (local address 127.0.0.1:36724)
0806 02:50:18.792358 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:18.792404 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:44889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260806 02:50:18.792610 3048 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.183.188:44889: Network error: Client connection negotiation failed: client connection to 127.1.183.188:44889: connect: Connection refused (error 111)
I20260806 02:50:18.849305 2922 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.183.188" port: 44889 } attempt: 2
I20260806 02:50:18.850286 2930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.849905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:44889 (local address 127.0.0.1:36728)
0806 02:50:18.850100 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:18.850181 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:44889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260806 02:50:18.851188 2930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.850934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:44889 (local address 127.0.0.1:36742)
0806 02:50:18.851048 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:18.851105 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:44889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 02:50:18.851382 2922 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.183.188:44889: Network error: Client connection negotiation failed: client connection to 127.1.183.188:44889: connect: Connection refused (error 111)
W20260806 02:50:18.856441 3059 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:50:18.856719 3059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:50:18.856796 3059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:50:18.856904 3059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 02:50:18.860762 3048 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.183.188" port: 44889 } attempt: 2
W20260806 02:50:18.861055 3059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:50:18.861157 3059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:50:18.861202 3059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:50:18.861238 3059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:50:18.861270 3059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:50:18.861647 3051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.861362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:44889 (local address 127.0.0.1:36756)
0806 02:50:18.861501 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:18.861572 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:44889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 02:50:18.862365 3051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.862149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:44889 (local address 127.0.0.1:36770)
0806 02:50:18.862259 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:18.862314 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.183.188:44889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 02:50:18.862495 3048 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.183.188:44889: Network error: Client connection negotiation failed: client connection to 127.1.183.188:44889: connect: Connection refused (error 111)
I20260806 02:50:18.866998 3059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.183.148:45647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.183.190:34181,127.1.183.189:33983,127.1.183.188:44889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/krb5kdc/kudu_127.1.183.188.keytab
--principal=kudu/127.1.183.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.183.188:44889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.183.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 02:05:18 UTC on ccycloud.quasar-smuers.root.comops.site
build id 4569538
I20260806 02:50:18.868139 3059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:50:18.869235 3059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:50:18.878106 3082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:18.878115 3077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:50:18.878103 3079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:50:18.878387 3059 server_base.cc:1034] running on GCE node
I20260806 02:50:18.879067 3059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:50:18.880211 3059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:50:18.881390 3059 hybrid_clock.cc:648] HybridClock initialized: now 1785984618881367 us; error 35 us; skew 500 ppm
Aug 06 02:50:18 dist-test-slave-gmjp krb5kdc[2814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984618, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:50:18.884372 3059 init.cc:377] Logged in from keytab as kudu/127.1.183.188@KRBTEST.COM (short username kudu)
I20260806 02:50:18.885329 3059 webserver.cc:460] Webserver started at http://127.1.183.188:39329/ using document root <none> and password file <none>
I20260806 02:50:18.885931 3059 fs_manager.cc:359] Metadata directory not provided
I20260806 02:50:18.885995 3059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:50:18.886221 3059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:50:18.888468 3059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-2/data/instance:
uuid: "a41cf7bd74a44b85a4413fcd232516a2"
format_stamp: "Formatted at 2026-08-06 02:50:18 on dist-test-slave-gmjp"
server_key: "4f62c0f6da177923bc05bfb20d52f3a4"
server_key_iv: "434720df0511b5c09ae9a374d0206ac7"
server_key_version: "encryptionkey@0"
I20260806 02:50:18.889081 3059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-2/wal/instance:
uuid: "a41cf7bd74a44b85a4413fcd232516a2"
format_stamp: "Formatted at 2026-08-06 02:50:18 on dist-test-slave-gmjp"
server_key: "4f62c0f6da177923bc05bfb20d52f3a4"
server_key_iv: "434720df0511b5c09ae9a374d0206ac7"
server_key_version: "encryptionkey@0"
I20260806 02:50:18.893016 3059 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:50:18.895403 3096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:50:18.896732 3059 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:50:18.896864 3059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-2/data,/tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-2/wal
uuid: "a41cf7bd74a44b85a4413fcd232516a2"
format_stamp: "Formatted at 2026-08-06 02:50:18 on dist-test-slave-gmjp"
server_key: "4f62c0f6da177923bc05bfb20d52f3a4"
server_key_iv: "434720df0511b5c09ae9a374d0206ac7"
server_key_version: "encryptionkey@0"
I20260806 02:50:18.896971 3059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:50:18.913718 3059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:50:18.917027 3059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:50:18.917236 3059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:50:18.924829 3059 rpc_server.cc:307] RPC server started. Bound to: 127.1.183.188:44889
I20260806 02:50:18.925135 3172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.183.188:44889 every 8 connection(s)
I20260806 02:50:18.925871 3059 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-2/data/info.pb
I20260806 02:50:18.928776 3173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:50:18.932606 3173 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34181 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:50:18.935000 1758 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hWVsi/build/debug/bin/kudu as pid 3059
I20260806 02:50:18.935166 1758 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hWVsi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984313887056-1758-0/minicluster-data/master-2/wal/instance
I20260806 02:50:18.935472 1758 external_mini_cluster.cc:1428] Setting key 6548eadcf03d5309962f95982778d98e
Aug 06 02:50:18 dist-test-slave-gmjp krb5kdc[2814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984618, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.188@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
Aug 06 02:50:18 dist-test-slave-gmjp krb5kdc[2814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.188@KRBTEST.COM
I20260806 02:50:18.947137 2927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.934469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48494 (local address 127.1.183.190:34181)
0806 02:50:18.934650 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:18.934655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:18.935567 (+ 912us) server_negotiation.cc:407] Connection header received
0806 02:50:18.937221 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:18.937224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:18.937271 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:18.937339 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:18.938320 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.938831 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.939956 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.940295 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.940379 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:18.943008 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:18.943019 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:18.943021 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:18.943048 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:18.944895 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:18.945518 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:18.945522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:18.945524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:18.945615 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:18.945959 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:18.945964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:18.945966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:18.946178 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:18.946287 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:18.946830 (+ 543us) server_negotiation.cc:299] Negotiation successful
0806 02:50:18.946960 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:50:18.948022 3175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.934856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.190:34181 (local address 127.0.0.1:48494)
0806 02:50:18.935401 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:18.935456 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:18.936996 (+ 1540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:18.937420 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:18.937429 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:18.937878 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:18.938166 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:18.938184 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.938960 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.938963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:18.939834 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:18.939844 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.940429 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.940433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:18.940606 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:18.941290 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:18.941305 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:18.942853 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:18.945026 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:18.945039 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:18.945052 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:18.945411 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:18.945710 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:18.945713 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:18.945715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:18.945852 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:18.946286 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:18.946294 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:18.946587 (+ 293us) client_negotiation.cc:769] Sending connection context
0806 02:50:18.946774 (+ 187us) client_negotiation.cc:240] Negotiation successful
0806 02:50:18.947107 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":1,"spinlock_wait_cycles":1280,"thread_start_us":167,"threads_started":1}
I20260806 02:50:18.948027 3177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.936188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36782 (local address 127.1.183.188:44889)
0806 02:50:18.936645 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:18.936660 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:18.936708 (+ 48us) server_negotiation.cc:407] Connection header received
0806 02:50:18.936996 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:18.937020 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:18.937281 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:18.937483 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:18.938203 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.939260 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.940233 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.940710 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.940829 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:18.943518 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:18.943545 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:18.943564 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:18.943592 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:18.945507 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:18.945937 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:18.945943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:18.945944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:18.946020 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:18.946303 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:18.946307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:18.946309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:18.946647 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:18.946877 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:18.947110 (+ 233us) server_negotiation.cc:299] Negotiation successful
0806 02:50:18.947257 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":333,"thread_start_us":128,"threads_started":1}
I20260806 02:50:18.949338 3173 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.183.189" port: 33983 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:50:18.950559 2922 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.183.188" port: 44889 } attempt: 3
I20260806 02:50:18.953656 3048 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.183.188" port: 44889 } attempt: 3
Aug 06 02:50:18 dist-test-slave-gmjp krb5kdc[2814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984618, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.190@KRBTEST.COM for kudu/127.1.183.188@KRBTEST.COM
Aug 06 02:50:18 dist-test-slave-gmjp krb5kdc[2814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984618, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.188@KRBTEST.COM for kudu/127.1.183.189@KRBTEST.COM
Aug 06 02:50:18 dist-test-slave-gmjp krb5kdc[2814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.190@KRBTEST.COM
I20260806 02:50:18.960356 2930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.950917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:44889 (local address 127.0.0.1:36786)
0806 02:50:18.951064 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:18.951077 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:18.951157 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:18.951351 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:18.951354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:18.951523 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:18.951611 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:18.951617 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.952693 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.952696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:18.953302 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:18.953312 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.953966 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.953970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:18.954058 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:18.954534 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:18.954548 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:18.956457 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:18.958592 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:18.958595 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:18.958598 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:18.958877 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:18.959192 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:18.959197 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:18.959199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:18.959301 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:18.959797 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:18.959801 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:18.959910 (+ 109us) client_negotiation.cc:769] Sending connection context
0806 02:50:18.960008 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 02:50:18.960135 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":4}
I20260806 02:50:18.961232 3177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.950975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36786 (local address 127.1.183.188:44889)
0806 02:50:18.951076 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:18.951080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:18.951123 (+ 43us) server_negotiation.cc:407] Connection header received
0806 02:50:18.951233 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:18.951235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:18.951267 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:18.951322 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:18.951714 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.952555 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.953444 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.953809 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.954277 (+ 468us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:18.956601 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:18.956606 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:18.956609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:18.956634 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:18.958425 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:18.958998 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:18.959001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:18.959003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:18.959090 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:18.959416 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:18.959420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:18.959421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:18.959658 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:18.960858 (+ 1200us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:18.960958 (+ 100us) server_negotiation.cc:299] Negotiation successful
0806 02:50:18.961074 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":39,"spinlock_wait_cycles":3968}
I20260806 02:50:18.961432 3057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.950051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48582 (local address 127.1.183.189:33983)
0806 02:50:18.950208 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:18.950213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:18.950228 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:50:18.950272 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:18.950276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:18.950324 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:18.950439 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:18.950913 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.951712 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.952808 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.953162 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.953268 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:18.956981 (+ 3713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:18.956985 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:18.956988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:18.957015 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:18.958837 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:18.959723 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:18.959727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:18.959729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:18.959813 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:18.960139 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:18.960146 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:18.960261 (+ 115us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:18.960507 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:18.960581 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:18.961123 (+ 542us) server_negotiation.cc:299] Negotiation successful
0806 02:50:18.961247 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:50:18.962002 2927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.950685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48510 (local address 127.1.183.190:34181)
0806 02:50:18.951810 (+ 1125us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:18.951814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:18.951824 (+ 10us) server_negotiation.cc:407] Connection header received
0806 02:50:18.951848 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:18.951850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:18.951884 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:18.951968 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:18.952675 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.953384 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.954530 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.954860 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.954979 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:18.957886 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:18.957889 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:18.957891 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:18.957912 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:18.959727 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:18.960272 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:18.960275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:18.960277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:18.960352 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:18.960669 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:18.960674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:18.960676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:18.960899 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:18.961103 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:18.961740 (+ 637us) server_negotiation.cc:299] Negotiation successful
0806 02:50:18.961852 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
Aug 06 02:50:18 dist-test-slave-gmjp krb5kdc[2814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984618, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.189@KRBTEST.COM for kudu/127.1.183.188@KRBTEST.COM
I20260806 02:50:18.964378 3175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.949907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.189:33983 (local address 127.0.0.1:48582)
0806 02:50:18.950109 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:18.950125 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:18.950198 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:18.950478 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:18.950484 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:18.950669 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:18.950798 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:18.950806 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.951810 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.951813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:18.952674 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:18.952683 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.953284 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.953287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:18.953404 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:18.954532 (+ 1128us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:18.954546 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:18.956807 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:18.959336 (+ 2529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:18.959339 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:18.959341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:18.959602 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:18.959931 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:18.959936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:18.959940 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:18.960025 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:18.960842 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:18.960845 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:18.960932 (+ 87us) client_negotiation.cc:769] Sending connection context
0806 02:50:18.961930 (+ 998us) client_negotiation.cc:240] Negotiation successful
0806 02:50:18.962260 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":36}
I20260806 02:50:18.964975 2922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdd7b9822ef2416ea8061a77c178ee2e: Bootstrap starting.
I20260806 02:50:18.967576 3051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.954046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:44889 (local address 127.0.0.1:36796)
0806 02:50:18.954231 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:18.954247 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:18.954323 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:18.957769 (+ 3446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:18.957775 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:18.959082 (+ 1307us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:18.959193 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:18.959202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.960737 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.960740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:18.961568 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:18.961583 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.962178 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.962182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:18.962263 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:18.962710 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:18.962724 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:18.964034 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:18.966057 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:18.966060 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:18.966063 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:18.966323 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:18.966643 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:18.966646 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:18.966648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:18.966737 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:18.967167 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:18.967170 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:18.967257 (+ 87us) client_negotiation.cc:769] Sending connection context
0806 02:50:18.967326 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 02:50:18.967422 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260806 02:50:18.967867 2922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdd7b9822ef2416ea8061a77c178ee2e: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:18.968385 3178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.954060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36796 (local address 127.1.183.188:44889)
0806 02:50:18.956768 (+ 2708us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:18.956777 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:18.956791 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:50:18.956829 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:18.956835 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:18.957171 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:18.958431 (+ 1260us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:18.959299 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.960609 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.961716 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.962038 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.962484 (+ 446us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:18.964193 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:18.964199 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:18.964205 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:18.964232 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:18.965941 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:18.966434 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:18.966438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:18.966440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:18.966530 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:18.966832 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:18.966836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:18.966837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:18.967046 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:18.967956 (+ 910us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:18.968042 (+ 86us) server_negotiation.cc:299] Negotiation successful
0806 02:50:18.968186 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2630,"thread_start_us":125,"threads_started":1}
I20260806 02:50:18.968781 2922 log.cc:824] T 00000000000000000000000000000000 P bdd7b9822ef2416ea8061a77c178ee2e: Log is configured to *not* fsync() on all Append() calls
Aug 06 02:50:18 dist-test-slave-gmjp krb5kdc[2814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.183.189@KRBTEST.COM
I20260806 02:50:18.971156 2922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdd7b9822ef2416ea8061a77c178ee2e: No bootstrap required, opened a new log
I20260806 02:50:18.971285 3173 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.183.188" port: 44889 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:50:18.972757 3048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6eb502e44df419fb3d82290a5dc912b: Bootstrap starting.
I20260806 02:50:18.974190 3057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.963373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48594 (local address 127.1.183.189:33983)
0806 02:50:18.964298 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:18.964302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:18.964315 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:50:18.964347 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:18.964349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:18.964388 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:18.964520 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:18.965408 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.966163 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.967141 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.967438 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.967538 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:18.970445 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:18.970448 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:18.970451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:18.970476 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:18.972102 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:18.972757 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:18.972761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:18.972763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:18.972848 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:18.973174 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:18.973177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:18.973179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:18.973380 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:18.973548 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:18.973927 (+ 379us) server_negotiation.cc:299] Negotiation successful
0806 02:50:18.974046 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":848}
I20260806 02:50:18.975553 3048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6eb502e44df419fb3d82290a5dc912b: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:18.976068 2922 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdd7b9822ef2416ea8061a77c178ee2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd7b9822ef2416ea8061a77c178ee2e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34181 } } peers { permanent_uuid: "a6eb502e44df419fb3d82290a5dc912b" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 33983 } } peers { permanent_uuid: "a41cf7bd74a44b85a4413fcd232516a2" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 44889 } }
I20260806 02:50:18.976341 2922 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdd7b9822ef2416ea8061a77c178ee2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdd7b9822ef2416ea8061a77c178ee2e, State: Initialized, Role: FOLLOWER
I20260806 02:50:18.976722 2922 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdd7b9822ef2416ea8061a77c178ee2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd7b9822ef2416ea8061a77c178ee2e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34181 } } peers { permanent_uuid: "a6eb502e44df419fb3d82290a5dc912b" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 33983 } } peers { permanent_uuid: "a41cf7bd74a44b85a4413fcd232516a2" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 44889 } }
Aug 06 02:50:18 dist-test-slave-gmjp krb5kdc[2814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984618, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.183.188@KRBTEST.COM for kudu/127.1.183.188@KRBTEST.COM
I20260806 02:50:18.977716 3048 log.cc:824] T 00000000000000000000000000000000 P a6eb502e44df419fb3d82290a5dc912b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:18.978335 2922 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdd7b9822ef2416ea8061a77c178ee2e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:18.978724 1758 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:50:18.980094 2891 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bdd7b9822ef2416ea8061a77c178ee2e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48510
I20260806 02:50:18.980409 2891 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdd7b9822ef2416ea8061a77c178ee2e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 02:50:18.980553 2891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdd7b9822ef2416ea8061a77c178ee2e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:18.981097 3175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.971828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.183.188:44889 (local address 127.0.0.1:36800)
0806 02:50:18.972062 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0806 02:50:18.972076 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:50:18.972176 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:50:18.972606 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:50:18.972609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:50:18.972775 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:50:18.972869 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:18.972877 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.973822 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.973825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:18.974594 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:50:18.974605 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.975216 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.975219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:50:18.975285 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:18.975763 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:50:18.975783 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:50:18.977302 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:50:18.979312 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:18.979317 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:18.979321 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:18.979563 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:18.979878 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:50:18.979882 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:50:18.979884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:50:18.979992 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:50:18.980564 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:50:18.980569 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:50:18.980658 (+ 89us) client_negotiation.cc:769] Sending connection context
0806 02:50:18.980775 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 02:50:18.980923 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170}
I20260806 02:50:18.981315 3178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.972264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36800 (local address 127.1.183.188:44889)
0806 02:50:18.972357 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:18.972361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:18.972371 (+ 10us) server_negotiation.cc:407] Connection header received
0806 02:50:18.972401 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:18.972404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:18.972443 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:18.972511 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:18.973001 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.973707 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.974716 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.975029 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.975121 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:18.977434 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:18.977437 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:18.977441 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:18.977469 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:18.979194 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:18.979665 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:18.979670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:18.979672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:18.979780 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:18.980111 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:18.980117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:18.980122 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:18.980443 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:18.980535 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:18.980866 (+ 331us) server_negotiation.cc:299] Negotiation successful
0806 02:50:18.980999 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260806 02:50:18.981977 2891 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdd7b9822ef2416ea8061a77c178ee2e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd7b9822ef2416ea8061a77c178ee2e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34181 } } peers { permanent_uuid: "a6eb502e44df419fb3d82290a5dc912b" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 33983 } } peers { permanent_uuid: "a41cf7bd74a44b85a4413fcd232516a2" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 44889 } }
I20260806 02:50:18.982823 3048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6eb502e44df419fb3d82290a5dc912b: No bootstrap required, opened a new log
I20260806 02:50:18.982805 3183 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdd7b9822ef2416ea8061a77c178ee2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd7b9822ef2416ea8061a77c178ee2e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34181 } } peers { permanent_uuid: "a6eb502e44df419fb3d82290a5dc912b" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 33983 } } peers { permanent_uuid: "a41cf7bd74a44b85a4413fcd232516a2" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 44889 } } }
I20260806 02:50:18.982935 3183 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdd7b9822ef2416ea8061a77c178ee2e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:50:18.982976 2891 leader_election.cc:288] T 00000000000000000000000000000000 P bdd7b9822ef2416ea8061a77c178ee2e [CANDIDATE]: Term 1 election: Requested vote from peers a6eb502e44df419fb3d82290a5dc912b (127.1.183.189:33983), a41cf7bd74a44b85a4413fcd232516a2 (127.1.183.188:44889)
I20260806 02:50:18.984270 3173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a41cf7bd74a44b85a4413fcd232516a2: Bootstrap starting.
I20260806 02:50:18.987100 3173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a41cf7bd74a44b85a4413fcd232516a2: Neither blocks nor log segments found. Creating new log.
I20260806 02:50:18.986933 3048 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6eb502e44df419fb3d82290a5dc912b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd7b9822ef2416ea8061a77c178ee2e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34181 } } peers { permanent_uuid: "a6eb502e44df419fb3d82290a5dc912b" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 33983 } } peers { permanent_uuid: "a41cf7bd74a44b85a4413fcd232516a2" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 44889 } }
I20260806 02:50:18.987174 3048 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6eb502e44df419fb3d82290a5dc912b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6eb502e44df419fb3d82290a5dc912b, State: Initialized, Role: FOLLOWER
I20260806 02:50:18.987524 3048 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6eb502e44df419fb3d82290a5dc912b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd7b9822ef2416ea8061a77c178ee2e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34181 } } peers { permanent_uuid: "a6eb502e44df419fb3d82290a5dc912b" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 33983 } } peers { permanent_uuid: "a41cf7bd74a44b85a4413fcd232516a2" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 44889 } }
I20260806 02:50:18.988025 3173 log.cc:824] T 00000000000000000000000000000000 P a41cf7bd74a44b85a4413fcd232516a2: Log is configured to *not* fsync() on all Append() calls
I20260806 02:50:18.989022 3048 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6eb502e44df419fb3d82290a5dc912b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:18.990195 3010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bdd7b9822ef2416ea8061a77c178ee2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a6eb502e44df419fb3d82290a5dc912b"
I20260806 02:50:18.990446 3010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6eb502e44df419fb3d82290a5dc912b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:18.991967 3010 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a6eb502e44df419fb3d82290a5dc912b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdd7b9822ef2416ea8061a77c178ee2e in term 1.
I20260806 02:50:18.992447 2859 leader_election.cc:302] T 00000000000000000000000000000000 P bdd7b9822ef2416ea8061a77c178ee2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6eb502e44df419fb3d82290a5dc912b, bdd7b9822ef2416ea8061a77c178ee2e; no voters:
I20260806 02:50:18.992934 3183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdd7b9822ef2416ea8061a77c178ee2e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:50:18.993207 3183 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdd7b9822ef2416ea8061a77c178ee2e [term 1 LEADER]: Becoming Leader. State: Replica: bdd7b9822ef2416ea8061a77c178ee2e, State: Running, Role: LEADER
I20260806 02:50:18.993613 3183 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdd7b9822ef2416ea8061a77c178ee2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd7b9822ef2416ea8061a77c178ee2e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34181 } } peers { permanent_uuid: "a6eb502e44df419fb3d82290a5dc912b" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 33983 } } peers { permanent_uuid: "a41cf7bd74a44b85a4413fcd232516a2" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 44889 } }
I20260806 02:50:18.995464 3183 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdd7b9822ef2416ea8061a77c178ee2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdd7b9822ef2416ea8061a77c178ee2e. Latest consensus state: current_term: 1 leader_uuid: "bdd7b9822ef2416ea8061a77c178ee2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd7b9822ef2416ea8061a77c178ee2e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34181 } } peers { permanent_uuid: "a6eb502e44df419fb3d82290a5dc912b" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 33983 } } peers { permanent_uuid: "a41cf7bd74a44b85a4413fcd232516a2" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 44889 } } }
I20260806 02:50:18.995579 3183 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdd7b9822ef2416ea8061a77c178ee2e [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:18.991888 2828 master_runner.cc:186] Error getting master registration for 127.1.183.189:33983: OK, instance_id { permanent_uuid: "a6eb502e44df419fb3d82290a5dc912b" instance_seqno: 1785984618698774 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 02:50:18.996210 2828 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:50:18.999614 3173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a41cf7bd74a44b85a4413fcd232516a2: No bootstrap required, opened a new log
I20260806 02:50:19.001879 3057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.985411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48610 (local address 127.1.183.189:33983)
0806 02:50:18.991171 (+ 5760us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:18.991176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:18.991188 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:50:18.991224 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:18.991228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:18.991274 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:18.991361 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:18.992436 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.993180 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.994596 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.994936 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.995034 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:18.997396 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:18.997402 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:18.997408 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:18.997452 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:18.999194 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:18.999702 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:18.999708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:18.999712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:18.999799 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:19.000491 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:19.000495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:19.000497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:19.000768 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:19.000875 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:19.001467 (+ 592us) server_negotiation.cc:299] Negotiation successful
0806 02:50:19.001662 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5669}
I20260806 02:50:19.002314 2927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.985457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48526 (local address 127.1.183.190:34181)
0806 02:50:18.985587 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:18.985591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:18.986328 (+ 737us) server_negotiation.cc:407] Connection header received
0806 02:50:18.986738 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:18.986744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:18.986801 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:18.986865 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:18.988949 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.989760 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.993351 (+ 3591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.993687 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.993765 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:18.995720 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:18.995725 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:18.995729 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:18.995759 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:18.997534 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:18.999574 (+ 2040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:18.999578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:18.999580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:18.999678 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:19.000606 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:19.000610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:19.000611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:19.000814 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:19.000938 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:19.002042 (+ 1104us) server_negotiation.cc:299] Negotiation successful
0806 02:50:19.002158 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 02:50:19.004068 3173 raft_consensus.cc:348] T 00000000000000000000000000000000 P a41cf7bd74a44b85a4413fcd232516a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd7b9822ef2416ea8061a77c178ee2e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34181 } } peers { permanent_uuid: "a6eb502e44df419fb3d82290a5dc912b" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 33983 } } peers { permanent_uuid: "a41cf7bd74a44b85a4413fcd232516a2" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 44889 } }
I20260806 02:50:19.004346 3173 raft_consensus.cc:729] T 00000000000000000000000000000000 P a41cf7bd74a44b85a4413fcd232516a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41cf7bd74a44b85a4413fcd232516a2, State: Initialized, Role: FOLLOWER
I20260806 02:50:19.004706 3173 consensus_queue.cc:260] T 00000000000000000000000000000000 P a41cf7bd74a44b85a4413fcd232516a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd7b9822ef2416ea8061a77c178ee2e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34181 } } peers { permanent_uuid: "a6eb502e44df419fb3d82290a5dc912b" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 33983 } } peers { permanent_uuid: "a41cf7bd74a44b85a4413fcd232516a2" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 44889 } }
I20260806 02:50:19.002292 2938 master_runner.cc:186] Error getting master registration for 127.1.183.188:44889: OK, instance_id { permanent_uuid: "a41cf7bd74a44b85a4413fcd232516a2" instance_seqno: 1785984618922646 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 02:50:19.005673 2938 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:50:19.006295 3173 sys_catalog.cc:564] T 00000000000000000000000000000000 P a41cf7bd74a44b85a4413fcd232516a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:50:19.006654 3233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260806 02:50:19.008915 3235 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a6eb502e44df419fb3d82290a5dc912b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:50:19.009065 3235 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:50:19.009631 3233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:50:19.006379 3140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bdd7b9822ef2416ea8061a77c178ee2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a41cf7bd74a44b85a4413fcd232516a2"
I20260806 02:50:19.007596 3178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:18.986405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36806 (local address 127.1.183.188:44889)
0806 02:50:18.986532 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:18.986538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:18.987303 (+ 765us) server_negotiation.cc:407] Connection header received
0806 02:50:18.987498 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:18.987501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:18.987553 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:18.987606 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:18.989057 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.989777 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.998044 (+ 8267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:18.998473 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:18.998573 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:19.003762 (+ 5189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:50:19.003766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:50:19.003769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:50:19.003796 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:50:19.005614 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:19.006201 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:19.006205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:19.006207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:19.006294 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:50:19.006603 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:50:19.006606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:50:19.006608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:50:19.006870 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:50:19.007042 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:19.007278 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 02:50:19.007413 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:50:19.011706 3140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a41cf7bd74a44b85a4413fcd232516a2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:50:19.013808 3140 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a41cf7bd74a44b85a4413fcd232516a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdd7b9822ef2416ea8061a77c178ee2e in term 1.
I20260806 02:50:19.015527 3140 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a41cf7bd74a44b85a4413fcd232516a2 [term 1 FOLLOWER]: Refusing update from remote peer bdd7b9822ef2416ea8061a77c178ee2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:19.015527 3010 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a6eb502e44df419fb3d82290a5dc912b [term 1 FOLLOWER]: Refusing update from remote peer bdd7b9822ef2416ea8061a77c178ee2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:50:19.016222 3230 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bdd7b9822ef2416ea8061a77c178ee2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6eb502e44df419fb3d82290a5dc912b" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 33983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:19.016490 3183 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bdd7b9822ef2416ea8061a77c178ee2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41cf7bd74a44b85a4413fcd232516a2" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 44889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:50:19.016800 3237 sys_catalog.cc:455] T 00000000000000000000000000000000 P a41cf7bd74a44b85a4413fcd232516a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd7b9822ef2416ea8061a77c178ee2e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34181 } } peers { permanent_uuid: "a6eb502e44df419fb3d82290a5dc912b" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 33983 } } peers { permanent_uuid: "a41cf7bd74a44b85a4413fcd232516a2" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 44889 } } }
I20260806 02:50:19.016968 3237 sys_catalog.cc:458] T 00000000000000000000000000000000 P a41cf7bd74a44b85a4413fcd232516a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:50:19.024334 3237 sys_catalog.cc:455] T 00000000000000000000000000000000 P a41cf7bd74a44b85a4413fcd232516a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdd7b9822ef2416ea8061a77c178ee2e. Latest consensus state: current_term: 1 leader_uuid: "bdd7b9822ef2416ea8061a77c178ee2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd7b9822ef2416ea8061a77c178ee2e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34181 } } peers { permanent_uuid: "a6eb502e44df419fb3d82290a5dc912b" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 33983 } } peers { permanent_uuid: "a41cf7bd74a44b85a4413fcd232516a2" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 44889 } } }
I20260806 02:50:19.024645 3237 sys_catalog.cc:458] T 00000000000000000000000000000000 P a41cf7bd74a44b85a4413fcd232516a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:50:19.027112 3217 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6eb502e44df419fb3d82290a5dc912b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bdd7b9822ef2416ea8061a77c178ee2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd7b9822ef2416ea8061a77c178ee2e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34181 } } peers { permanent_uuid: "a6eb502e44df419fb3d82290a5dc912b" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 33983 } } peers { permanent_uuid: "a41cf7bd74a44b85a4413fcd232516a2" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 44889 } } }
I20260806 02:50:19.027266 3217 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6eb502e44df419fb3d82290a5dc912b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:50:19.027367 3217 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6eb502e44df419fb3d82290a5dc912b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdd7b9822ef2416ea8061a77c178ee2e. Latest consensus state: current_term: 1 leader_uuid: "bdd7b9822ef2416ea8061a77c178ee2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd7b9822ef2416ea8061a77c178ee2e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34181 } } peers { permanent_uuid: "a6eb502e44df419fb3d82290a5dc912b" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 33983 } } peers { permanent_uuid: "a41cf7bd74a44b85a4413fcd232516a2" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 44889 } } }
I20260806 02:50:19.027454 3217 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6eb502e44df419fb3d82290a5dc912b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:50:19.032004 3230 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdd7b9822ef2416ea8061a77c178ee2e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bdd7b9822ef2416ea8061a77c178ee2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd7b9822ef2416ea8061a77c178ee2e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34181 } } peers { permanent_uuid: "a6eb502e44df419fb3d82290a5dc912b" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 33983 } } peers { permanent_uuid: "a41cf7bd74a44b85a4413fcd232516a2" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 44889 } } }
I20260806 02:50:19.032135 3230 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdd7b9822ef2416ea8061a77c178ee2e [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:19.032438 3241 mvcc.cc:205] Tried to move back new op lower bound from 7315392999478185984 to 7315392999401394176. Current Snapshot: MvccSnapshot[applied={T|T < 7315392999478185984}]
I20260806 02:50:19.033097 3183 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdd7b9822ef2416ea8061a77c178ee2e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bdd7b9822ef2416ea8061a77c178ee2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd7b9822ef2416ea8061a77c178ee2e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34181 } } peers { permanent_uuid: "a6eb502e44df419fb3d82290a5dc912b" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 33983 } } peers { permanent_uuid: "a41cf7bd74a44b85a4413fcd232516a2" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 44889 } } }
I20260806 02:50:19.033196 3183 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdd7b9822ef2416ea8061a77c178ee2e [sys.catalog]: This master's current role is: LEADER
I20260806 02:50:19.033887 3233 catalog_manager.cc:1347] Generated new cluster ID: f0b01ef4ac904b1b821140c7da964057
I20260806 02:50:19.033946 3233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:50:19.034648 3237 sys_catalog.cc:455] T 00000000000000000000000000000000 P a41cf7bd74a44b85a4413fcd232516a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bdd7b9822ef2416ea8061a77c178ee2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd7b9822ef2416ea8061a77c178ee2e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34181 } } peers { permanent_uuid: "a6eb502e44df419fb3d82290a5dc912b" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 33983 } } peers { permanent_uuid: "a41cf7bd74a44b85a4413fcd232516a2" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 44889 } } }
I20260806 02:50:19.034754 3237 sys_catalog.cc:458] T 00000000000000000000000000000000 P a41cf7bd74a44b85a4413fcd232516a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:50:19.037055 3246 mvcc.cc:205] Tried to move back new op lower bound from 7315392999478185984 to 7315392999401394176. Current Snapshot: MvccSnapshot[applied={T|T < 7315392999478185984}]
I20260806 02:50:19.037201 3244 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6eb502e44df419fb3d82290a5dc912b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bdd7b9822ef2416ea8061a77c178ee2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd7b9822ef2416ea8061a77c178ee2e" member_type: VOTER last_known_addr { host: "127.1.183.190" port: 34181 } } peers { permanent_uuid: "a6eb502e44df419fb3d82290a5dc912b" member_type: VOTER last_known_addr { host: "127.1.183.189" port: 33983 } } peers { permanent_uuid: "a41cf7bd74a44b85a4413fcd232516a2" member_type: VOTER last_known_addr { host: "127.1.183.188" port: 44889 } } }
I20260806 02:50:19.037302 3244 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6eb502e44df419fb3d82290a5dc912b [sys.catalog]: This master's current role is: FOLLOWER
W20260806 02:50:19.038688 3263 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a41cf7bd74a44b85a4413fcd232516a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:50:19.038756 3263 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:50:19.040238 3243 mvcc.cc:205] Tried to move back new op lower bound from 7315392999478185984 to 7315392999401394176. Current Snapshot: MvccSnapshot[applied={T|T < 7315392999478185984 or (T in {7315392999478185984})}]
I20260806 02:50:19.052826 3233 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:50:19.054342 3233 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:50:19.065992 3233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdd7b9822ef2416ea8061a77c178ee2e: Generated new TSK 0
I20260806 02:50:19.066881 3233 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:50:19.098310 3057 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:19.095165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48622 (local address 127.1.183.189:33983)
0806 02:50:19.095327 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:19.095333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:19.097159 (+ 1826us) server_negotiation.cc:407] Connection header received
0806 02:50:19.097359 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:19.097363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:19.097417 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:19.097474 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:19.098163 (+ 689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48622: BlockingRecv error: recv got EOF from 127.0.0.1:48622 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260806 02:50:19.100004 2927 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:19.095572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48528 (local address 127.1.183.190:34181)
0806 02:50:19.095703 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:19.095708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:19.099048 (+ 3340us) server_negotiation.cc:407] Connection header received
0806 02:50:19.099226 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:19.099230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:19.099282 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:19.099333 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:19.099856 (+ 523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48528: BlockingRecv error: recv got EOF from 127.0.0.1:48528 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260806 02:50:19.101651 3178 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:19.096368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36822 (local address 127.1.183.188:44889)
0806 02:50:19.096508 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:19.096513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:19.100701 (+ 4188us) server_negotiation.cc:407] Connection header received
0806 02:50:19.100880 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:19.100884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:19.100939 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:19.100989 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:19.101512 (+ 523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36822: BlockingRecv error: recv got EOF from 127.0.0.1:36822 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260806 02:50:19.106103 3057 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:19.104721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48628 (local address 127.1.183.189:33983)
0806 02:50:19.104843 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:19.104847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:19.105109 (+ 262us) server_negotiation.cc:407] Connection header received
0806 02:50:19.105339 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:19.105342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:19.105430 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:19.105484 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:19.105995 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48628: BlockingRecv error: recv got EOF from 127.0.0.1:48628 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260806 02:50:19.111460 3057 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:19.109929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48642 (local address 127.1.183.189:33983)
0806 02:50:19.110037 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:19.110041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:19.110556 (+ 515us) server_negotiation.cc:407] Connection header received
0806 02:50:19.110735 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:19.110738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:19.110789 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:19.110838 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:19.111355 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48642: BlockingRecv error: recv got EOF from 127.0.0.1:48642 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260806 02:50:19.117151 3057 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:19.115773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48654 (local address 127.1.183.189:33983)
0806 02:50:19.115899 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:19.115903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:19.116219 (+ 316us) server_negotiation.cc:407] Connection header received
0806 02:50:19.116441 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:19.116445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:19.116493 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:19.116543 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:19.117053 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48654: BlockingRecv error: recv got EOF from 127.0.0.1:48654 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260806 02:50:19.125303 3057 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:19.123611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48662 (local address 127.1.183.189:33983)
0806 02:50:19.123762 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:19.123766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:19.124351 (+ 585us) server_negotiation.cc:407] Connection header received
0806 02:50:19.124560 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:19.124564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:19.124613 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:19.124663 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:19.125196 (+ 533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48662: BlockingRecv error: recv got EOF from 127.0.0.1:48662 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260806 02:50:19.141573 3057 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:19.135958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48664 (local address 127.1.183.189:33983)
0806 02:50:19.136089 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:19.136093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:19.140501 (+ 4408us) server_negotiation.cc:407] Connection header received
0806 02:50:19.140727 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:19.140732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:19.140790 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:19.140844 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:19.141442 (+ 598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48664: BlockingRecv error: recv got EOF from 127.0.0.1:48664 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260806 02:50:19.162535 3057 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:19.160624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48666 (local address 127.1.183.189:33983)
0806 02:50:19.160765 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:19.160769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:19.161409 (+ 640us) server_negotiation.cc:407] Connection header received
0806 02:50:19.161637 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:19.161641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:19.161709 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:19.161768 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:19.162408 (+ 640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48666: BlockingRecv error: recv got EOF from 127.0.0.1:48666 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260806 02:50:19.199739 3057 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:19.197914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48668 (local address 127.1.183.189:33983)
0806 02:50:19.198054 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:19.198059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:19.198709 (+ 650us) server_negotiation.cc:407] Connection header received
0806 02:50:19.198917 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:19.198932 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:19.198988 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:19.199040 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:19.199616 (+ 576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48668: BlockingRecv error: recv got EOF from 127.0.0.1:48668 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260806 02:50:19.271713 3057 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:19.270548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48670 (local address 127.1.183.189:33983)
0806 02:50:19.270715 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:19.270720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:19.270881 (+ 161us) server_negotiation.cc:407] Connection header received
0806 02:50:19.271019 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:19.271022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:19.271071 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:19.271144 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:19.271584 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48670: BlockingRecv error: recv got EOF from 127.0.0.1:48670 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260806 02:50:19.403456 3057 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:19.402155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48678 (local address 127.1.183.189:33983)
0806 02:50:19.402332 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:19.402338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:19.402488 (+ 150us) server_negotiation.cc:407] Connection header received
0806 02:50:19.402673 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:19.402681 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:19.402733 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:19.402824 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:19.403319 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48678: BlockingRecv error: recv got EOF from 127.0.0.1:48678 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260806 02:50:19.662984 3057 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:50:19.661733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48690 (local address 127.1.183.189:33983)
0806 02:50:19.661884 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:19.661890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:19.661992 (+ 102us) server_negotiation.cc:407] Connection header received
0806 02:50:19.662175 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:19.662178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:19.662236 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:19.662332 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:50:19.662859 (+ 527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48690: BlockingRecv error: recv got EOF from 127.0.0.1:48690 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 02:50:20.010635 3235 catalog_manager.cc:1259] Loaded cluster ID: f0b01ef4ac904b1b821140c7da964057
I20260806 02:50:20.010732 3235 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a6eb502e44df419fb3d82290a5dc912b: loading cluster ID for follower catalog manager: success
I20260806 02:50:20.012403 3235 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a6eb502e44df419fb3d82290a5dc912b: acquiring CA information for follower catalog manager: success
I20260806 02:50:20.013561 3235 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a6eb502e44df419fb3d82290a5dc912b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:50:20.040125 3263 catalog_manager.cc:1259] Loaded cluster ID: f0b01ef4ac904b1b821140c7da964057
I20260806 02:50:20.040247 3263 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a41cf7bd74a44b85a4413fcd232516a2: loading cluster ID for follower catalog manager: success
I20260806 02:50:20.041855 3263 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a41cf7bd74a44b85a4413fcd232516a2: acquiring CA information for follower catalog manager: success
I20260806 02:50:20.042991 3263 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a41cf7bd74a44b85a4413fcd232516a2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:50:20.181231 3367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:20.177217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48706 (local address 127.1.183.189:33983)
0806 02:50:20.177473 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:20.177477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:20.177496 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:50:20.177629 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:20.177632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:20.177766 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:20.177956 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:50:20.178201 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:20.178757 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:20.179698 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:20.179961 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:20.180060 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:20.180359 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:50:20.180810 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:50:20.180902 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:20.180970 (+ 68us) server_negotiation.cc:299] Negotiation successful
0806 02:50:20.181025 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":82,"threads_started":1}
I20260806 02:50:20.187923 3378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:20.184484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36830 (local address 127.1.183.188:44889)
0806 02:50:20.184729 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:20.184733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:20.184745 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:50:20.184826 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:20.184828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:20.184947 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:20.185044 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:50:20.185230 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:20.185729 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:20.186516 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:20.186732 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:20.186807 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:20.187076 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:50:20.187509 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:50:20.187597 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:20.187653 (+ 56us) server_negotiation.cc:299] Negotiation successful
0806 02:50:20.187699 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":99,"threads_started":1}
I20260806 02:50:20.195220 3367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:20.191082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48716 (local address 127.1.183.189:33983)
0806 02:50:20.191210 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:20.191214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:20.191496 (+ 282us) server_negotiation.cc:407] Connection header received
0806 02:50:20.191733 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:20.191736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:20.191793 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:20.191881 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:50:20.192122 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:20.192967 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:20.194112 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:20.194358 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:20.194470 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:20.194787 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:50:20.194875 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:50:20.194937 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:20.195024 (+ 87us) server_negotiation.cc:299] Negotiation successful
0806 02:50:20.195078 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:50:20.196034 3388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:20.191117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48542 (local address 127.1.183.190:34181)
0806 02:50:20.191366 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:20.191371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:20.191486 (+ 115us) server_negotiation.cc:407] Connection header received
0806 02:50:20.191658 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:20.191662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:20.191805 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:20.191933 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:50:20.192319 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:20.193112 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:20.194166 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:20.194535 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:20.194642 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:20.194992 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:50:20.195390 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:50:20.195498 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:20.195706 (+ 208us) server_negotiation.cc:299] Negotiation successful
0806 02:50:20.195770 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":64,"threads_started":1}
I20260806 02:50:20.196112 3378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:50:20.191568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36838 (local address 127.1.183.188:44889)
0806 02:50:20.191674 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0806 02:50:20.191679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:50:20.192385 (+ 706us) server_negotiation.cc:407] Connection header received
0806 02:50:20.192499 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:50:20.192502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:50:20.192550 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:50:20.192625 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:50:20.192877 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:20.193594 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:20.194817 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:50:20.195147 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:50:20.195218 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:50:20.195630 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:50:20.195707 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:50:20.195753 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0806 02:50:20.195921 (+ 168us) server_negotiation.cc:299] Negotiation successful
0806 02:50:20.195973 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 02:50:20.199414 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 2828
I20260806 02:50:20.220700 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 2938
I20260806 02:50:20.234692 1758 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hWVsi/build/debug/bin/kudu with pid 3059
2026-08-06T02:50:20Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3314 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6703 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (306347 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST